This question already has answers here:
Editing specific line in text file in Python
(11 answers)
Closed 3 months ago.
Is there a way, in Python, to modify a single line in a file without a for loop looping through all the lines?
The exact positions within the file that need to be modified are unknown.
This should work -
f = open(r'full_path_to_your_file', 'r') # pass an appropriate path of the required file
lines = f.readlines()
lines[n-1] = "your new text for this line" # n is the line number you want to edit; subtract 1 as indexing of list starts from 0
f.close() # close the file and reopen in write mode to enable writing to file; you can also open in append mode and use "seek", but you will have some unwanted old data if the new data is shorter in length.
f = open(r'full_path_to_your_file', 'w')
f.writelines(lines)
# do the remaining operations on the file
f.close()
However, this can be resource consuming (both time and memory) if your file size is too large, because the f.readlines() function loads the entire file, split into lines, in a list.
This will be just fine for small and medium sized files.
Unless we're talking about a fairly contrived situation in which you already know a lot about the file, the answer is no. You have to iterate over the file to determine where the newline characters are; there's nothing special about a "line" when it comes to file storage -- it all looks the same.
Yes, you can modify the line in place, but if the length changes, you will have to rewrite the remainder of the file.
You'll also need to know where the line is, in the file. This usually means the program needs to at least read through the file up to the line that needs to be changed.
There are exceptions - if the lines are all fixed length, or you have some sort of index on the file for example
Related
This question already has answers here:
Insert line at middle of file with Python?
(11 answers)
Closed 4 years ago.
I'm trying to insert some text at specific position of file using this:
with open("test.txt","r+") as f:
f.seek(5)
f.write("B")
But this overwrites character at position 5 with new data ("B") instead of inserting it.
for example if i have
AAAAAAAAAA
in file test.txt and run the code
I get AAAAABAAAA instead of AAAAABAAAAA (five A must be after B)
How can i insert at desired position of file instead of overwrite?
There are three answers for that:
Generic file API (one you expect on all OSes) have no interface for 'insert' (read: this is impossible)
You can implement this by yourself, by reading whole file into memory, composing new content and writing it back. (If file is big, you may need to create some code to do this in chunks).
Good news for linux users: Since linux 3.15 it's possible to insert holes in the middle of the file (basically, shifting everything in file starting from specific location of a specific offset). There is a comprehensive article on this topic here: https://lwn.net/Articles/629965/. It is supported for ext4 and XFS filesystems, and it requires some low-level operations on fd (e.f. not the usual open for the python). Moreover, as I checked (Sep 2018) a fallocate module on pypi does not support it, so you need to write a low-level code to do FALLOC_FL_INSERT_RANGE ioctl.
TL;DR; If you file is small, read it into memory and do insert in memory. If you file is medium size (1-2Gb) do it in temp file and rename it after that. If your file is large, use windowed operations or dig down to FALLOC_FL_INSERT_RANGE (if you have a relatively modern linux).
This worked for me :
with open("test.txt","r+") as f:
f.seek(5) #first fseek to the position
line=f.readline() #read everything after it
f.seek(5) #since file pointer has moved, fseek back to 5
f.write("B") #write the letter
f.write(line) #write the remaining part
Original : AAAAAAAAAA
After : AAAAABAAAAA
f1 = open("test.txt","r+")
f1.seek(5)
data = "{}{}".format("B",f1.readline())
f1.seek(5)
f1.write(data)
f1.close()
This question already has answers here:
How to read a large file - line by line?
(11 answers)
Closed 6 years ago.
I have a Python script which needs to read a section of a very large text file, starting at line N and ending at N+X.
I don't want to use "open('file')", because that will write the entire thing to the memory, which will both take too long, and waste too much memory.
My script runs on a Unix machine, so I currently use the native head and tail functions, i.e.:
section = subprocess.check_output('tail -n-N {filePath} | head -n X')
but is feels like there must be a smarter way of doing it..
is there a way to get lines N through N+X of a text file in Python without opening the entire file?
Thanks!
Python's islice() works well for doing this:
from itertools import islice
N = 2
X = 5
with open('large_file.txt') as f_input:
for row in islice(f_input, N-1, N+X):
print row.strip()
This skips over all of the initial lines and just returns the lines you are interested in.
The answer to your question is located here: How to read large file, line by line in python
with open(...) as f:
for line in f:
<do something with line>
The with statement handles opening and closing the file, including if
an exception is raised in the inner block. The for line in f treats
the file object f as an iterable, which automatically uses buffered IO
and memory management so you don't have to worry about large files.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Editing specific line in text file in python
I am writing a software that allows users to write data into a text file. However, I am not sure how to delete the first line of the text file and rewrite the line. I want the user to be able to update the text file's first line by clicking on a button and inputing in something but that requires deleting and writing a new line as the first line which I am not sure how to implement. Any help would be appreciated.
Edit:
So I sought out the first line of the file and tried to write another line but that doesn't delete the previous line.
file.seek(0)
file.write("This is the new first line \n")
You did not describe how you opened the file to begin with. If you used file = open(somename, "a") that file will not be truncated but new data is written at the end (even after a seek on most if not all modern systems). You would have to open the file with "r+")
But your example assumes that the line you write is exactly the same length as what the user typed. There is no line organisation in the files, just bytes, some of which indicate line ending.
Wat you need to do is use a temporary file or a temporary buffer in memory for all the lines and then write the lines out with the first replaced.
If things fit in memory (which I assume since few users are going to type so much it does not fit), you should be able to do:
lines = open(somename, 'r').readlines()
lines[0] = "This is the new first line \n"
file = open(somename, 'w')
for line in lines:
file.write(line)
file.close()
You could use readlines to get an array of lines and then use del on the first index of the array. This might help. http://www.daniweb.com/software-development/python/threads/68765/how-to-remove-a-number-of-lines-from-a-text-file-
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Python: How to read huge text file into memory
To process a large text file(1G+) line by line , random access by any line number is desired, most importantly, without loading the whole file content into RAM. Is there a python library to do that?
It is beneficial when analyzing a large log file, read only is enough.
If there is no such standard library, I have to seek an alternative method: Find a set of function/class that can return the N-th line of sub-string from a big string-like object, so that I can mmap(yes, I mean memory-mapped file object) the file to that object then do line-based processing.
Thank you.
PS: A log file is almost sure to have variable line length.
I think that something like below might work, since the file object's method readline() reads one line at a time. If the lines are of arbitrary length, you need to index the positions like follows.
lines = [0]
with open("testmat.txt") as f:
while f.readline():
lines.append(f.tell())
# now you can read an arbitrary line:
f.seek(lines[1235])
line = f.readline()
If the lines were of same length, you could just do f.seek(linenumber*linelenght)
This question already has answers here:
How to read a file line-by-line into a list?
(28 answers)
Closed 8 years ago.
I want to prompt a user for a number of random numbers to be generated and saved to a file. He gave us that part. The part we have to do is to open that file, convert the numbers into a list, then find the mean, standard deviation, etc. without using the easy built-in Python tools.
I've tried using open but it gives me invalid syntax (the file name I chose was "numbers" and it saved into "My Documents" automatically, so I tried open(numbers, 'r') and open(C:\name\MyDocuments\numbers, 'r') and neither one worked).
with open('C:/path/numbers.txt') as f:
lines = f.read().splitlines()
this will give you a list of values (strings) you had in your file, with newlines stripped.
also, watch your backslashes in windows path names, as those are also escape chars in strings. You can use forward slashes or double backslashes instead.
Two ways to read file into list in python (note these are not either or) -
use of with - supported from python 2.5 and above
use of list comprehensions
1. use of with
This is the pythonic way of opening and reading files.
#Sample 1 - elucidating each step but not memory efficient
lines = []
with open("C:\name\MyDocuments\numbers") as file:
for line in file:
line = line.strip() #or some other preprocessing
lines.append(line) #storing everything in memory!
#Sample 2 - a more pythonic and idiomatic way but still not memory efficient
with open("C:\name\MyDocuments\numbers") as file:
lines = [line.strip() for line in file]
#Sample 3 - a more pythonic way with efficient memory usage. Proper usage of with and file iterators.
with open("C:\name\MyDocuments\numbers") as file:
for line in file:
line = line.strip() #preprocess line
doSomethingWithThisLine(line) #take action on line instead of storing in a list. more memory efficient at the cost of execution speed.
the .strip() is used for each line of the file to remove \n newline character that each line might have. When the with ends, the file will be closed automatically for you. This is true even if an exception is raised inside of it.
2. use of list comprehension
This could be considered inefficient as the file descriptor might not be closed immediately. Could be a potential issue when this is called inside a function opening thousands of files.
data = [line.strip() for line in open("C:/name/MyDocuments/numbers", 'r')]
Note that file closing is implementation dependent. Normally unused variables are garbage collected by python interpreter. In cPython (the regular interpreter version from python.org), it will happen immediately, since its garbage collector works by reference counting. In another interpreter, like Jython or Iron Python, there may be a delay.
f = open("file.txt")
lines = f.readlines()
Look over here. readlines() returns a list containing one line per element. Note that these lines contain the \n (newline-character) at the end of the line. You can strip off this newline-character by using the strip()-method. I.e. call lines[index].strip() in order to get the string without the newline character.
As joaquin noted, do not forget to f.close() the file.
Converting strint to integers is easy: int("12").
The pythonic way to read a file and put every lines in a list:
from __future__ import with_statement #for python 2.5
with open('C:/path/numbers.txt', 'r') as f:
lines = f.readlines()
Then, assuming that each lines contains a number,
numbers =[int(e.strip()) for e in lines]
You need to pass a filename string to open. There's an extra complication when the string has \ in it, because that's a special string escape character to Python. You can fix this by doubling up each as \\ or by putting a r in front of the string as follows: r'C:\name\MyDocuments\numbers'.
Edit: The edits to the question make it completely different from the original, and since none of them was from the original poster I'm not sure they're warrented. However it does point out one obvious thing that might have been overlooked, and that's how to add "My Documents" to a filename.
In an English version of Windows XP, My Documents is actually C:\Documents and Settings\name\My Documents. This means the open call should look like:
open(r"C:\Documents and Settings\name\My Documents\numbers", 'r')
I presume you're using XP because you call it My Documents - it changed in Vista and Windows 7. I don't know if there's an easy way to look this up automatically in Python.
hdl = open("C:/name/MyDocuments/numbers", 'r')
milist = hdl.readlines()
hdl.close()
To summarize a bit from what people have been saying:
f=open('data.txt', 'w') # will make a new file or erase a file of that name if it is present
f=open('data.txt', 'r') # will open a file as read-only
f=open('data.txt', 'a') # will open a file for appending (appended data goes to the end of the file)
If you wish have something in place similar to a try/catch
with open('data.txt') as f:
for line in f:
print line
I think #movieyoda code is probably what you should use however
If you have multiple numbers per line and you have multiple lines, you can read them in like this:
#!/usr/bin/env python
from os.path import dirname
with open(dirname(__file__) + '/data/path/filename.txt') as input_data:
input_list= [map(int,num.split()) for num in input_data.readlines()]