Develop Reference

splitlines() and iterating over an opened file give different results

I have files with sometimes weird end-of-lines characters like \r\r\n. With this, it works like I want:
with open('test.txt', 'wb') as f: # simulate a file with weird end-of-lines
f.write(b'abc\r\r\ndef')
with open('test.txt', 'rb') as f:
for l in f:
print(l)
# b'abc\r\r\n'
# b'def'
I want to able to get the same result from a string. I thought about splitlines but it does not give the same result:
print(b'abc\r\r\ndef'.splitlines())
# [b'abc', b'', b'def']
Even with keepends=True, it's not the same result.
Question: how to have the same behaviour than for l in f with splitlines()?
Linked: Changing str.splitlines to match file readlines and https://bugs.python.org/issue22232
Note: I don't want to put everything in a BytesIO or StringIO, because it does a x0.5 speed performance (already benchmarked); I want to keep a simple string. So it's not a duplicate of How do I wrap a string in a file in Python?.

Why don't you just split it:
input = b'\nabc\r\r\r\nd\ref\nghi\r\njkl'
result = input.split(b'\n')
print(result)
[b'', b'abc\r\r\r', b'd\ref', b'ghi\r', b'jkl']
You will loose the trailing \n that can be added later to every line, if you really need them. On the last line there is a need to check if it is really needed. Like
fixed = [bstr + b'\n' for bstr in result]
if input[-1] != b'\n':
fixed[-1] = fixed[-1][:-1]
print(fixed)
[b'\n', b'abc\r\r\r\n', b'd\ref\n', b'ghi\r\n', b'jkl']
Another variant with a generator. This way it will be memory savvy on the huge files and the syntax will be similar to the original for l in bin_split(input) :
def bin_split(input_str):
start = 0
while start>=0 :
found = input_str.find(b'\n', start) + 1
if 0 < found < len(input_str):
yield input_str[start : found]
start = found
else:
yield input_str[start:]
break

There are a couple ways to do this, but none are especially fast.
If you want to keep the line endings, you might try the re module:
lines = re.findall(r'[\r\n]+|[^\r\n]+[\r\n]*', text)
# or equivalently
line_split_regex = re.compile(r'[\r\n]+|[^\r\n]+[\r\n]*')
lines = line_split_regex.findall(text)
If you need the endings and the file is really big, you may want to iterate instead:
for r in re.finditer(r'[\r\n]+|[^\r\n]+[\r\n]*', text):
line = r.group()
# do stuff with line here
If you don't need the endings, then you can do it much more easily:
lines = list(filter(None, text.splitlines()))
You can omit the list() part if you just iterate over the results (or if using Python2):
for line in filter(None, text.splitlines()):
pass # do stuff with line

I would iterate through like this:
text = "b'abc\r\r\ndef'"
results = text.split('\r\r\n')
for r in results:
print(r)

This is a for l in f: solution:
The key to this is the newline argument on the open call. From the documentation:
[![enter image description here][1]][1]
Therefore, you should use newline='' when writing to suppress newline translation and then when reading use newline='\n', which will work if all your lines terminate with 0 or more '\r' characters followed by a '\n' character:
with open('test.txt', 'w', newline='') as f:
f.write('abc\r\r\ndef')
with open('test.txt', 'r', newline='\n') as f:
for line in f:
print(repr(line))
Prints:
'abc\r\r\n'
'def'
A quasi-splitlines solution:
This strictly speaking not a splitlines solution since to be able to handle arbitrary line endings a regular expression version of split would have to be used capturing the line endings and then re-assembling the lines and their endings. So, instead this solution just uses a regular expression to break up the input text allowing line endings consisting of any number of '\r' characters followed by a '\n' character:
import re
input = '\nabc\r\r\ndef\nghi\r\njkl'
with open('test.txt', 'w', newline='') as f:
f.write(input)
with open('test.txt', 'r', newline='') as f:
text = f.read()
lines = re.findall(r'[^\r\n]*\r*\n|[^\r\n]+$', text)
for line in lines:
print(repr(line))
Prints:
'\n'
'abc\r\r\n'
'def\n'
'ghi\r\n'
'jkl'
Regex Demo

how to replace a line of two words in a file using python

I want to replace a line in a file but my code doesn't do what I want. The code doesn't change that line. It seems that the problem is the space between ALS and 4277 characters in the input.txt. I need to keep that space in the file. How can I fix my code?
A part part of input.txt:
ALS 4277
Related part of the code:
for lines in fileinput.input('input.txt', inplace=True):
print(lines.rstrip().replace("ALS"+str(4277), "KLM" + str(4945)))
Desired output:
KLM 4945

Using the same idea that other user have already pointed out, you could also reproduce the same spacing, by first matching the spacing and saving it in a variable (spacing in my code):
import re
with open('input.txt') as f:
lines = f.read()
match = re.match(r'ALS(\s+)4277', lines)
if match != None:
spacing = match.group(1)
lines = re.sub(r'ALS\s+4277', 'KLM%s4945'%spacing, lines.rstrip())
print lines

As the spaces vary you will need to use regex to account for the spaces.
import re
lines = "ALS 4277 "
line = re.sub(r"(ALS\s+4277)", "KLM 4945", lines.rstrip())
print(line)

Try:
with open('input.txt') as f:
for line in f:
a, b = line.strip().split()
if a == 'ALS' and b == '4277':
line = line.replace(a, 'KLM').replace(b, '4945')
print(line, end='') # as line has '\n'

Python: How to count average length of lines in a text file

I'm having trouble defining a function that takes a filename as a parameter and returns the average line length. Particularly having trouble removing the "\n" in the file.
This is what I have so far:
def averageLineLength(fn):
fn = open("Ex_file.txt", "r")
lines = fn.readlines()
return (sum(len(line) for line in lines) / len(lines))

You can use strip() to remove preceding and trailing \n from the line and minor modification to your own code should be sufficient
def averageLineLength(fn):
fn = open("Ex_file.txt", "r")
lines = fn.readlines()
return sum([len(line.strip('\n')) for line in lines]) / len(lines)
This cleans out all leading and trailing whitespace. If you only want to strip out \n at the end of the line
(sum(len(line.replace("\n",'')) for line in lines) / len(lines))

The common way to get rid of whitespace (such as newlines) in python is strip. There's also rstrip if you want to preserve the left side, and you can give any of them an argument if you only want to target newlines specifically:
>>> ' Hello, world \n'.strip()
'Hello, world'
>>> ' Hello, world \n'.rstrip()
' Hello, world'
>>> ' Hello, world \n'.strip('\n')
' Hello, world '
Two other notes: The original function did not actually use the filename, and in Python 2 it also performs integer division (which may or may not be intentional). With these modifications:
def averageLineLength(fn):
with open(fn) as f:
lines = [line.strip() for line in f]
return 1.0 * sum(map(len, lines)) / len(lines)

What you have already will tell you the average line length.
There are a few different methods for handling removal of the '\n'.
The simplest is to just use the "strip" method. This will remove all leading and trailing whitespace from each line.
If you want to remove only the trailing '\n'. You can write a simple list comprehension like this to do so.
[l[:-1] if l[-1] == "\n" else l for l in lines]
Or simply remove the final character without checking, trusting it is a '\n' because you used "readlines".
[l[:-1] for l in lines]
You should also use the "with" construction block on your file to ensure it is closed when the program exits the block. Making these changes, your function becomes the following:
def averageLineLength(fn):
with open("Ex_file.txt", "r") as fn:
lines = [l.strip() for l in fn.readlines()]
return (sum(len(line) for line in lines) / len(lines)
Or alternatively, if you want to preserve leading and trailing whitespace that is not '\n':
def averageLineLength(fn):
with open("Ex_file.txt", "r") as fn:
lines = [l[:-1] for l in fn.readlines()]
return (sum(len(line) for line in lines) / len(lines)

This solution should solve the problem as well:
def averageLineLength(fn):
with open('Ex_file.txt".txt', 'r') as fn:
lst = fn.readlines()
return sum([len(line.strip()) for line in lst]) / len(lst)

How to remove all whitespace and newlines?

Assuming I have a file that contains the following:
Assume <tab> is actually a tab and <space> is actually a space. (ignore quotes)
"
<tab><tab>
<space>
<tab>
The clothes at
the superstore are
at a discount today.
"
Assume this is in a text file. How do I remove all the spaces such that the resulting text file is (ignore the quotes:
"
The clothes at
the superstore are
at a discount today.
"

Try this, assuming you don't want to overwrite the old file. Easy to adapt if you do:
oldfile = open("EXISTINGFILENAME", "r")
data = oldfile.read()
oldfile.close()
stripped_data = data.lstrip()
newfile = open("NEWFILENAME", "w")
newfile.write(stripped_data)
newfile.close()
Note that this will only remove leading whitespace, to remove any trailing whitespace as well, use strip in place of lstrip.

Something like this perhaps (don't know if you need a python solution or if cmdline-tools are ok):
$ cat -t INPUT
^I^I
^I^I
"^I
^I^I^I
^I ghi
"
$ sed '/^[ ]*$/d' INPUT
"
ghi
"
I.e. remove lines only containing spaces/and/or tabs as well as empty limes.

If you want to preserve indentation and trailing space on the lines in your output file, test the stripped line, but write the raw line.
This also uses context managers, and works in Python 2.7:
with open('EXISTINGFILE', 'r') as fin, open('NEWFILE', 'w') as fout:
for line in fin:
if line.strip():
fout.write(line)
If you want to do other processing, I'd suggest defining that in its own function body, and calling that function:
def process_line(line):
# for example
return ''.join(('Payload:\t', line.strip().upper(), '\tEnd Payload\n'))
with open('EXISTINGFILE', 'r') as fin, open('NEWFILE', 'w') as fout:
for line in fin:
if line.strip():
fout.write(process_line(line))
Rereading your question, I see that you only asked about removing whitespace at the beginning of your file. If you want to get EVERY line of your source file after a certain condition is met, you can set a flag for that condition, and switch your output based on the flag.
For example, if you want to remove initial lines of whitespace, process non-whitespace lines, and not remove or process all whitespace lines after you have at least one line of data, you could do this:
def process_line(line):
# for example
return ''.join(('Payload:\t', line.strip().upper(), '\tEnd Payload\n'))
with open('EXISTINGFILE', 'r') as fin, open('NEWFILE', 'w') as fout:
have_paydata = False
for line in fin:
if line.strip():
have_paydata = True if not have_paydata
fout.write(process_line(line))
elif have_paydata:
fout.write(line)

strip() removes all leading/trailing whitespace, then after we do that test if there are any characters left in the line:
with f as open("file.txt", "r"):
for line in f:
if len(line.strip()):
print line

lstrip will remove all whitespace from the beginning of a string. If you need to keep the leading whitespace on the first text line, use a regex instead:
import re
data = '''\
\t\t
\t
The clothes at
the superstore are
at a discount today.
'''
# Remove ALL whitespace from the start of string
print(data.lstrip())
# Remove all whitespace from start of string up to and including a newline
print(re.sub(r'^\s*\n',r'',data))
Output:
The clothes at
the superstore are
at a discount today.
The clothes at
the superstore are
at a discount today.
To modify a file this way:
# A with statement closes the file on exit from the block
with open('data.txt') as f:
data = f.read()
data = re.sub(r'^\s*\n',r'',data))
with open('data.txt','w') as f:
f.write(data)

How to read a text file into a string variable and strip newlines?

I have a text file that looks like:
ABC
DEF
How can I read the file into a single-line string without newlines, in this case creating a string 'ABCDEF'?
For reading the file into a list of lines, but removing the trailing newline character from each line, see How to read a file without newlines?.

You could use:
with open('data.txt', 'r') as file:
data = file.read().replace('\n', '')
Or if the file content is guaranteed to be one-line
with open('data.txt', 'r') as file:
data = file.read().rstrip()

In Python 3.5 or later, using pathlib you can copy text file contents into a variable and close the file in one line:
from pathlib import Path
txt = Path('data.txt').read_text()
and then you can use str.replace to remove the newlines:
txt = txt.replace('\n', '')

You can read from a file in one line:
str = open('very_Important.txt', 'r').read()
Please note that this does not close the file explicitly.
CPython will close the file when it exits as part of the garbage collection.
But other python implementations won't. To write portable code, it is better to use with or close the file explicitly. Short is not always better. See https://stackoverflow.com/a/7396043/362951

To join all lines into a string and remove new lines, I normally use :
with open('t.txt') as f:
s = " ".join([l.rstrip("\n") for l in f])

with open("data.txt") as myfile:
data="".join(line.rstrip() for line in myfile)
join() will join a list of strings, and rstrip() with no arguments will trim whitespace, including newlines, from the end of strings.

This can be done using the read() method :
text_as_string = open('Your_Text_File.txt', 'r').read()
Or as the default mode itself is 'r' (read) so simply use,
text_as_string = open('Your_Text_File.txt').read()

I'm surprised nobody mentioned splitlines() yet.
with open ("data.txt", "r") as myfile:
data = myfile.read().splitlines()
Variable data is now a list that looks like this when printed:
['LLKKKKKKKKMMMMMMMMNNNNNNNNNNNNN', 'GGGGGGGGGHHHHHHHHHHHHHHHHHHHHEEEEEEEE']
Note there are no newlines (\n).
At that point, it sounds like you want to print back the lines to console, which you can achieve with a for loop:
for line in data:
print(line)

It's hard to tell exactly what you're after, but something like this should get you started:
with open ("data.txt", "r") as myfile:
data = ' '.join([line.replace('\n', '') for line in myfile.readlines()])

I have fiddled around with this for a while and have prefer to use use read in combination with rstrip. Without rstrip("\n"), Python adds a newline to the end of the string, which in most cases is not very useful.
with open("myfile.txt") as f:
file_content = f.read().rstrip("\n")
print(file_content)

Here are four codes for you to choose one:
with open("my_text_file.txt", "r") as file:
data = file.read().replace("\n", "")
or
with open("my_text_file.txt", "r") as file:
data = "".join(file.read().split("\n"))
or
with open("my_text_file.txt", "r") as file:
data = "".join(file.read().splitlines())
or
with open("my_text_file.txt", "r") as file:
data = "".join([line for line in file])

you can compress this into one into two lines of code!!!
content = open('filepath','r').read().replace('\n',' ')
print(content)
if your file reads:
hello how are you?
who are you?
blank blank
python output
hello how are you? who are you? blank blank

You can also strip each line and concatenate into a final string.
myfile = open("data.txt","r")
data = ""
lines = myfile.readlines()
for line in lines:
data = data + line.strip();
This would also work out just fine.

This is a one line, copy-pasteable solution that also closes the file object:
_ = open('data.txt', 'r'); data = _.read(); _.close()

f = open('data.txt','r')
string = ""
while 1:
line = f.readline()
if not line:break
string += line
f.close()
print(string)

python3: Google "list comprehension" if the square bracket syntax is new to you.
with open('data.txt') as f:
lines = [ line.strip('\n') for line in list(f) ]

Oneliner:
List: "".join([line.rstrip('\n') for line in open('file.txt')])
Generator: "".join((line.rstrip('\n') for line in open('file.txt')))
List is faster than generator but heavier on memory. Generators are slower than lists and is lighter for memory like iterating over lines. In case of "".join(), I think both should work well. .join() function should be removed to get list or generator respectively.
Note: close() / closing of file descriptor probably not needed

Have you tried this?
x = "yourfilename.txt"
y = open(x, 'r').read()
print(y)

To remove line breaks using Python you can use replace function of a string.
This example removes all 3 types of line breaks:
my_string = open('lala.json').read()
print(my_string)
my_string = my_string.replace("\r","").replace("\n","")
print(my_string)
Example file is:
{
"lala": "lulu",
"foo": "bar"
}
You can try it using this replay scenario:
https://repl.it/repls/AnnualJointHardware

I don't feel that anyone addressed the [ ] part of your question. When you read each line into your variable, because there were multiple lines before you replaced the \n with '' you ended up creating a list. If you have a variable of x and print it out just by
x
or print(x)
or str(x)
You will see the entire list with the brackets. If you call each element of the (array of sorts)
x[0]
then it omits the brackets. If you use the str() function you will see just the data and not the '' either.
str(x[0])

Maybe you could try this? I use this in my programs.
Data= open ('data.txt', 'r')
data = Data.readlines()
for i in range(len(data)):
data[i] = data[i].strip()+ ' '
data = ''.join(data).strip()

Regular expression works too:
import re
with open("depression.txt") as f:
l = re.split(' ', re.sub('\n',' ', f.read()))[:-1]
print (l)
['I', 'feel', 'empty', 'and', 'dead', 'inside']

with open('data.txt', 'r') as file:
data = [line.strip('\n') for line in file.readlines()]
data = ''.join(data)

from pathlib import Path
line_lst = Path("to/the/file.txt").read_text().splitlines()
Is the best way to get all the lines of a file, the '\n' are already stripped by the splitlines() (which smartly recognize win/mac/unix lines types).
But if nonetheless you want to strip each lines:
line_lst = [line.strip() for line in txt = Path("to/the/file.txt").read_text().splitlines()]
strip() was just a useful exemple, but you can process your line as you please.
At the end, you just want concatenated text ?
txt = ''.join(Path("to/the/file.txt").read_text().splitlines())

This works:
Change your file to:
LLKKKKKKKKMMMMMMMMNNNNNNNNNNNNN GGGGGGGGGHHHHHHHHHHHHHHHHHHHHEEEEEEEE
Then:
file = open("file.txt")
line = file.read()
words = line.split()
This creates a list named words that equals:
['LLKKKKKKKKMMMMMMMMNNNNNNNNNNNNN', 'GGGGGGGGGHHHHHHHHHHHHHHHHHHHHEEEEEEEE']
That got rid of the "\n". To answer the part about the brackets getting in your way, just do this:
for word in words: # Assuming words is the list above
print word # Prints each word in file on a different line
Or:
print words[0] + ",", words[1] # Note that the "+" symbol indicates no spaces
#The comma not in parentheses indicates a space
This returns:
LLKKKKKKKKMMMMMMMMNNNNNNNNNNNNN, GGGGGGGGGHHHHHHHHHHHHHHHHHHHHEEEEEEEE

with open(player_name, 'r') as myfile:
data=myfile.readline()
list=data.split(" ")
word=list[0]
This code will help you to read the first line and then using the list and split option you can convert the first line word separated by space to be stored in a list.
Than you can easily access any word, or even store it in a string.
You can also do the same thing with using a for loop.

file = open("myfile.txt", "r")
lines = file.readlines()
str = '' #string declaration
for i in range(len(lines)):
str += lines[i].rstrip('\n') + ' '
print str

Try the following:
with open('data.txt', 'r') as myfile:
data = myfile.read()
sentences = data.split('\\n')
for sentence in sentences:
print(sentence)
Caution: It does not remove the \n. It is just for viewing the text as if there were no \n