This question already has answers here:
Insert line at middle of file with Python?
(11 answers)
Closed 4 years ago.
I'm trying to insert some text at specific position of file using this:
with open("test.txt","r+") as f:
f.seek(5)
f.write("B")
But this overwrites character at position 5 with new data ("B") instead of inserting it.
for example if i have
AAAAAAAAAA
in file test.txt and run the code
I get AAAAABAAAA instead of AAAAABAAAAA (five A must be after B)
How can i insert at desired position of file instead of overwrite?
There are three answers for that:
Generic file API (one you expect on all OSes) have no interface for 'insert' (read: this is impossible)
You can implement this by yourself, by reading whole file into memory, composing new content and writing it back. (If file is big, you may need to create some code to do this in chunks).
Good news for linux users: Since linux 3.15 it's possible to insert holes in the middle of the file (basically, shifting everything in file starting from specific location of a specific offset). There is a comprehensive article on this topic here: https://lwn.net/Articles/629965/. It is supported for ext4 and XFS filesystems, and it requires some low-level operations on fd (e.f. not the usual open for the python). Moreover, as I checked (Sep 2018) a fallocate module on pypi does not support it, so you need to write a low-level code to do FALLOC_FL_INSERT_RANGE ioctl.
TL;DR; If you file is small, read it into memory and do insert in memory. If you file is medium size (1-2Gb) do it in temp file and rename it after that. If your file is large, use windowed operations or dig down to FALLOC_FL_INSERT_RANGE (if you have a relatively modern linux).
This worked for me :
with open("test.txt","r+") as f:
f.seek(5) #first fseek to the position
line=f.readline() #read everything after it
f.seek(5) #since file pointer has moved, fseek back to 5
f.write("B") #write the letter
f.write(line) #write the remaining part
Original : AAAAAAAAAA
After : AAAAABAAAAA
f1 = open("test.txt","r+")
f1.seek(5)
data = "{}{}".format("B",f1.readline())
f1.seek(5)
f1.write(data)
f1.close()
Related
This question already has answers here:
Python readlines() usage and efficient practice for reading
(2 answers)
Closed 7 years ago.
I am new to computer science and am trying to create a function in python that will open files on my computer.
I know that the function f.readline() grabs the current line as a string, but what makes the functions f.read() and for line in f: different? Thanks.
read(x) will read up to x bytes in a file. If you don't supply the size, the entire file is read.
readline(x) will read up to x bytes or a newline, whichever comes first. If you don't supply a size, it will read all data until it hits a newline.
When using for line in f, it will call the next() method under the hood which really just does something very similar to readline (although I see references that is may do some buffering more efficiently since iterating usually means you are planning to read the entire file).
There is also readlines() which reads all lines into memory.
This question already has answers here:
Unable to read huge (20GB) file from CPython
(2 answers)
Closed 9 years ago.
I'm a Python newbie and had a quick question regarding memory usage when reading large text files. I have a ~13GB csv I'm trying to read line-by-line following the Python documentation and more experienced Python user's advice to not use readlines() in order to avoid loading the entire file into memory.
When trying to read a line from the file I get the error below and am not sure what might be causing it. Besides this error, I also notice my PC's memory usage is excessively high. This was a little surprising since my understanding of the readline function is that it only loads a single line from the file at a time into memory.
For reference, I'm using Continuum Analytic's Anaconda distribution of Python 2.7 and PyScripter as my IDE for debugging and testing. Any help or insight is appreciated.
with open(R'C:\temp\datasets\a13GBfile.csv','r') as f:
foo = f.readline(); #<-- Err: SystemError: ..\Objects\stringobject.c:3902 bad argument to internal function
UPDATE:
Thank you all for the quick, informative and very helpful feedback, I reviewed the referenced link which is exactly the problem I was having. After applying the documented 'rU' option mode I was able to read lines from the file like normal. I didn't notice this mode mentioned in the documentation link I was referencing initially and neglected to look at the details for the open function first. Thanks again.
Unix text files end each line with \n.
Windows text files end each line with \r\n.
When you open a file in text mode, 'r', Python assumes it has the native line endings for your platform.
So, if you open a Unix text file on Windows, Python will look for \r\n sequences to split the lines. But there won't be any, so it'll treat your whole file is one giant 13-billion-character line. So that readline() call ends up trying to read the whole thing into memory.
The fix for this is to use universal newlines mode, by opening the file in mode rU. As explained in the docs for open:
supplying 'U' opens the file as a text file, but lines may be terminated by any of the following: the Unix end-of-line convention '\n', the Macintosh convention '\r', or the Windows convention '\r\n'.
So, instead of searching for \r\n sequences to split the lines, it looks for \r\n, or \n, or \r. And there are millions of \n. So, the problem is solved.
A different way to fix this is to use binary mode, 'rb'. In this mode, Python doesn't do any conversion at all, and assumes all lines end in \n, no matter what platform you're on.
On its own, this is pretty hacky—it means you'll end up with an extra \r on the end of every line in a Windows text file.
But it means you can pass the file on to a higher-level file reader like csv that wants binary files, so it can parse them the way it wants to. On top of magically solving this problem for you, a higher-level library will also probably make the rest of your code a lot simpler and more robust. For example, it might look something like this:
with open(R'C:\temp\datasets\a13GBfile.csv','rb') as f:
for row in csv.reader(f):
# do stuff
Now each row is automatically split on commas, except that commas that are inside quotes or escaped in the appropriate way don't count, and so on, so all you need to deal with is a list of column values.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Python: How to read huge text file into memory
To process a large text file(1G+) line by line , random access by any line number is desired, most importantly, without loading the whole file content into RAM. Is there a python library to do that?
It is beneficial when analyzing a large log file, read only is enough.
If there is no such standard library, I have to seek an alternative method: Find a set of function/class that can return the N-th line of sub-string from a big string-like object, so that I can mmap(yes, I mean memory-mapped file object) the file to that object then do line-based processing.
Thank you.
PS: A log file is almost sure to have variable line length.
I think that something like below might work, since the file object's method readline() reads one line at a time. If the lines are of arbitrary length, you need to index the positions like follows.
lines = [0]
with open("testmat.txt") as f:
while f.readline():
lines.append(f.tell())
# now you can read an arbitrary line:
f.seek(lines[1235])
line = f.readline()
If the lines were of same length, you could just do f.seek(linenumber*linelenght)
This question already has answers here:
Editing specific line in text file in Python
(11 answers)
Closed 3 months ago.
Is there a way, in Python, to modify a single line in a file without a for loop looping through all the lines?
The exact positions within the file that need to be modified are unknown.
This should work -
f = open(r'full_path_to_your_file', 'r') # pass an appropriate path of the required file
lines = f.readlines()
lines[n-1] = "your new text for this line" # n is the line number you want to edit; subtract 1 as indexing of list starts from 0
f.close() # close the file and reopen in write mode to enable writing to file; you can also open in append mode and use "seek", but you will have some unwanted old data if the new data is shorter in length.
f = open(r'full_path_to_your_file', 'w')
f.writelines(lines)
# do the remaining operations on the file
f.close()
However, this can be resource consuming (both time and memory) if your file size is too large, because the f.readlines() function loads the entire file, split into lines, in a list.
This will be just fine for small and medium sized files.
Unless we're talking about a fairly contrived situation in which you already know a lot about the file, the answer is no. You have to iterate over the file to determine where the newline characters are; there's nothing special about a "line" when it comes to file storage -- it all looks the same.
Yes, you can modify the line in place, but if the length changes, you will have to rewrite the remainder of the file.
You'll also need to know where the line is, in the file. This usually means the program needs to at least read through the file up to the line that needs to be changed.
There are exceptions - if the lines are all fixed length, or you have some sort of index on the file for example
This question already has answers here:
How to read a file line-by-line into a list?
(28 answers)
Closed 8 years ago.
I want to prompt a user for a number of random numbers to be generated and saved to a file. He gave us that part. The part we have to do is to open that file, convert the numbers into a list, then find the mean, standard deviation, etc. without using the easy built-in Python tools.
I've tried using open but it gives me invalid syntax (the file name I chose was "numbers" and it saved into "My Documents" automatically, so I tried open(numbers, 'r') and open(C:\name\MyDocuments\numbers, 'r') and neither one worked).
with open('C:/path/numbers.txt') as f:
lines = f.read().splitlines()
this will give you a list of values (strings) you had in your file, with newlines stripped.
also, watch your backslashes in windows path names, as those are also escape chars in strings. You can use forward slashes or double backslashes instead.
Two ways to read file into list in python (note these are not either or) -
use of with - supported from python 2.5 and above
use of list comprehensions
1. use of with
This is the pythonic way of opening and reading files.
#Sample 1 - elucidating each step but not memory efficient
lines = []
with open("C:\name\MyDocuments\numbers") as file:
for line in file:
line = line.strip() #or some other preprocessing
lines.append(line) #storing everything in memory!
#Sample 2 - a more pythonic and idiomatic way but still not memory efficient
with open("C:\name\MyDocuments\numbers") as file:
lines = [line.strip() for line in file]
#Sample 3 - a more pythonic way with efficient memory usage. Proper usage of with and file iterators.
with open("C:\name\MyDocuments\numbers") as file:
for line in file:
line = line.strip() #preprocess line
doSomethingWithThisLine(line) #take action on line instead of storing in a list. more memory efficient at the cost of execution speed.
the .strip() is used for each line of the file to remove \n newline character that each line might have. When the with ends, the file will be closed automatically for you. This is true even if an exception is raised inside of it.
2. use of list comprehension
This could be considered inefficient as the file descriptor might not be closed immediately. Could be a potential issue when this is called inside a function opening thousands of files.
data = [line.strip() for line in open("C:/name/MyDocuments/numbers", 'r')]
Note that file closing is implementation dependent. Normally unused variables are garbage collected by python interpreter. In cPython (the regular interpreter version from python.org), it will happen immediately, since its garbage collector works by reference counting. In another interpreter, like Jython or Iron Python, there may be a delay.
f = open("file.txt")
lines = f.readlines()
Look over here. readlines() returns a list containing one line per element. Note that these lines contain the \n (newline-character) at the end of the line. You can strip off this newline-character by using the strip()-method. I.e. call lines[index].strip() in order to get the string without the newline character.
As joaquin noted, do not forget to f.close() the file.
Converting strint to integers is easy: int("12").
The pythonic way to read a file and put every lines in a list:
from __future__ import with_statement #for python 2.5
with open('C:/path/numbers.txt', 'r') as f:
lines = f.readlines()
Then, assuming that each lines contains a number,
numbers =[int(e.strip()) for e in lines]
You need to pass a filename string to open. There's an extra complication when the string has \ in it, because that's a special string escape character to Python. You can fix this by doubling up each as \\ or by putting a r in front of the string as follows: r'C:\name\MyDocuments\numbers'.
Edit: The edits to the question make it completely different from the original, and since none of them was from the original poster I'm not sure they're warrented. However it does point out one obvious thing that might have been overlooked, and that's how to add "My Documents" to a filename.
In an English version of Windows XP, My Documents is actually C:\Documents and Settings\name\My Documents. This means the open call should look like:
open(r"C:\Documents and Settings\name\My Documents\numbers", 'r')
I presume you're using XP because you call it My Documents - it changed in Vista and Windows 7. I don't know if there's an easy way to look this up automatically in Python.
hdl = open("C:/name/MyDocuments/numbers", 'r')
milist = hdl.readlines()
hdl.close()
To summarize a bit from what people have been saying:
f=open('data.txt', 'w') # will make a new file or erase a file of that name if it is present
f=open('data.txt', 'r') # will open a file as read-only
f=open('data.txt', 'a') # will open a file for appending (appended data goes to the end of the file)
If you wish have something in place similar to a try/catch
with open('data.txt') as f:
for line in f:
print line
I think #movieyoda code is probably what you should use however
If you have multiple numbers per line and you have multiple lines, you can read them in like this:
#!/usr/bin/env python
from os.path import dirname
with open(dirname(__file__) + '/data/path/filename.txt') as input_data:
input_list= [map(int,num.split()) for num in input_data.readlines()]