I'm working on Windows. I've a Python file to create a new CSV file and I view that using Notepad (even through Microsoft Excel).
import csv
data = [['fruit','quantity'], ['apple',5], ['banana',7],['mango',8]]
with open('d:\lineter.csv', 'w') as l:
w = csv.writer(l,delimiter='|', lineterminator='\r')
w.writerows(data)
The resulting file in Notepad:
fruit|quantityapple|5banana|7mango|8
Does the carriage return \r work or not? It works like lineterminator='' in Notepad. But in Excel, it works like '\n'.
The output doesn't seem to implement carriage return. When I use lineterminator as:
w = csv.writer(l, delimiter='|', lineterminator='*\r*\n')
The output in Notepad is:
fruit|quantity**
apple|5**
banana|7**
mango|8**
This is evident here too.
How does '\r' work in lineterminator in writer()?
Or is there another thing happening there?
The shorter answer:
When to use carriage return (CR, \r) vs. line feed (LF, \n) vs. both (CRLF, \r\n) to make a new line appear in a text editor on Windows, Mac, and Linux:
How does '\r' work in lineterminator in writer()??
It works fine in csv.writer(). This really isn't a Python, CSV, or writer problem. This is an operating system historical difference (actually, it's more accurate to state it is a program-specific difference) going back to the 1960s or so.
Or is there another thing happening there?
Yes, this is the one.
Your version of Notepad doesn't recognize a carriage return (\r) as a character used to display new lines, and hence won't display it as such in Notepad. Other text editors, such as Sublime Text 3, however probably would, even on Windows.
Up until about the year 2018 or so, Windows and Notepad required a carriage return + line feed (\r\n) together to display a new line. Contrast this to Mac and Linux, which require only \n.
The solution is to use \r\n for a new line on Windows, and \n alone for a new line on Mac or Linux. You can also try a different text editor, such as Sublime Text, when viewing or editing text files, or upgrade your version of Windows or Notepad, if possible, as somewhere around the year 2018 Windows Notepad started to accept \r alone as a valid old-Mac-style new line char.
(from the OP's comment under this answer):
Then why to give '\r\n'???
When a programmer writes a program, the programmer can make the program do whatever the programmer wants the program to do. When Windows programmers made Windows and Notepad they decided to make the program do nothing if it got a \r, nothing if it got a \n, and to do a new line if it got a \r\n together. It's that simple. The program is doing exactly what the programmers told it to do, because they decided that's how they wanted the program to work. So, if you want a new line in the older (pre-2018) version of Notepad in Windows, you must do what the programmers require you to do to get it. \r\n is it.
This goes back to the days of teletypewriters (read the "History" and "Representation" sections here), and this page about "teleprinters" / "teletypewriters" / "teletype or TTY machines" too:
A typewriter or electromechanical printer can print characters on paper, and execute operations such as move the carriage back to the left margin of the same line (carriage return), advance to the same column of the next line (line feed), and so on.
(source; emphasis added)
The mechanical carriage return button on a teletypewriter (\r now on a computer) meant: "return the carriage (print head) to the beginning of the line" (meaning: the far left side of the page), and the line feed mechanical mechanism on a teletypewriter (\n now on a computer) meant: "roll the paper up one line so we can now type onto the next line." Without the mechanical line feed (\n) action, the carriage return (\r) alone would move the mechanical print head to the far left of the page and cause you to type right back on top of the words you already typed! And without the carriage return mechanical action (\r on a computer), the line feed mechanical action (\n) alone would cause you to just type in the last column at the far right on each new line on the page, never able to return the print head to the left side of the page again! On an electro-mechanical teletypewriter, they both had to be used: the carriage return would bring the print head back to the left side of the page, and the line feed action would move the print head down to the next line. So, presumably, Windows programmers felt it was logical to keep that tradition alive, and they decided to require both a \r\n together to create a new line on a computer, since that's how it had to be done traditionally on an electro-mechanical teletypewriter.
Read below for details.
Details (the longer answer):
I have some ideas of what's going on, but let's take a look. I believe we have two questions to answer:
Is the \r actually being stored into the file?
Is Notepad actually showing the \r, and if not, why not?
So, for #1. Let's test it on Linux Ubuntu 20.04 (Focal Fossa):
This program:
#!/usr/bin/python3
import csv
data = [['fruit','quantity'], ['apple',5], ['banana',7],['mango',8]]
with open('d:\lineter.csv','w') as l:
w = csv.writer(l, delimiter='|', lineterminator='\r')
w.writerows(data)
produces this file: d:\lineter.csv. If I open it in the Sublime Text 3 text editor I see:
fruit|quantity
apple|5
banana|7
mango|8
So far so good. Let's look at the characters with hexdump at the command line:
hexdump -c shows the \r characters, sure enough!
$ hexdump -c d\:\\lineter.csv
0000000 f r u i t | q u a n t i t y \r a
0000010 p p l e | 5 \r b a n a n a | 7 \r
0000020 m a n g o | 8 \r
0000028
You can also use hexdump -C to show the characters in hexadecimal instead, and again, I see the \r in the file as a hex 0d char, which is correct.
Ok, so I boot up Windows 10 Professional in my VirtualBox virtual machine in Linux, and open the same file in Notepad, and....it works too! See screenshot:
But, notice the part I circled which says "Macintosh (CR)". I'm running the latest version of Windows 10 Professional. I'm betting you're using an old version of Notepad which doesn't have this fix, and yours won't say that here. This is because for 33 years Notepad didn't handle Carriage Return, or \r, as a valid line-ending, so it wouldn't display it as such. See here: Windows Notepad fixed after 33 years: Now it finally handles Unix, Mac OS line endings.
Due to historical differences dating back to teletypewriters and Morse code (read the "History" and "Representation" sections here), different systems decided to make their text editors treat line endings in different ways. From the article just above (emphasis added):
Notepad previously recognized only the Windows End of Line (EOL) characters, specifically Carriage Return (CR, \r, 0x0d) and Line Feed (LF, \n, 0x0a) together.
For old-school Mac OS, the EOL character is just Carriage Return (CR, \r, 0x0d) and for Linux/Unix it's just Line Feed (LF, \n, 0x0a). Modern macOS, since Mac OS X, follows the Unix convention.
So, what we have here is the following displayed as a newline in a text editor:
Old-school Mac: CR (\r) only
Windows Notepad up until ~2018: CR & LF together (\r\n)
Linux: LF (\n) only
Modern Mac: LF (\n) only
Modern Windows Notepad (year ~2018 and later): any of the scenarios above.
So, for Windows, just stick to always using \r\n for a newline, and for Mac or Linux, just stick to always using \n for a newline, unless you're trying to guarantee old-school (i.e., pre-2019 :)) Windows compatibility of your files, in which case you should use \r\n for newlines as well.
Note, for Sublime Text 3, I just searched the preferences in Preferences → Settings and found this setting:
// Determines what character(s) are used to terminate each line in new files.
// Valid values are 'system' (whatever the OS uses), 'windows' (CRLF) and
// 'unix' (LF only).
"default_line_ending": "system",
So, to use the convention for whatever OS you're running Sublime Text on, the default is "system". To force 'windows' (CRLF) line endings when editing and saving files in Sublime Text, however, use this:
"default_line_ending": "windows",
And to force Unix (Mac and Linux) LF-only line ending settings, use this:
"default_line_ending": "unix",
In the Notepad editor, I can find no such settings to configure. It is a simple editor, catering for 33 years to Windows line endings only.
Additional Reading:
https://en.wikipedia.org/wiki/Teleprinter
https://en.wikipedia.org/wiki/Newline#History
Is a new line = \n OR \r\n?
Why does Windows use CR LF?
[I still need to read & study] Unix & Linux: Why does Linux use LF as the newline character?
[I still need to read & study] Retrocomputing: Why is Windows using CR+LF and Unix just LF when Unix is the older system?
I am trying to solve Kattis problem. The full problem is found in the link: https://open.kattis.com/problems/addingwords
The part of the problem that I'm confused with is : "Input is a sequence of up to 2000 commands, one per line, ending at end of file."
What would be the code for this input? I tried doing this:
import sys
for line in sys.stdin.readlines():
#print('something')
After this, I continued the program as normal within the indentation from above. My question is, how would I test whether the program is working in cmd? I want to test a few cases, but when I input something, the command prompt keeps waiting for other results instead of printing anything out. And when I press control C the program ends abruptly. How are we supposed to check whether the program is working while taking in user input till end of file?
The problem here is that readlines() is eager not lazy. That means it will read the entire file into memory (until EOF) and then split it into lines and return a list of those lines. So when working with interactive stdin, sys.stdin.readline() will wait until the end of stdin (Ctrl-D on linux/macOS, Ctrl+Z on windows).
But there is no need for readlines() (and in fact, you almost should never use it). Iterating over a file object by default does so by lines:
for line in sys.stdin:
print('got line!')
The docs even admit that you should do this. If you do need all lines in a list, then just do list(sys.stdin).
I have an input file I'm trying to process with Python, which appears to have content like the following:
# This works, when run at a REPL
line = 'aababasdf|75=2|asdfa|150=17|asdfasdf'
date = line.split('|75=')[1].split('|',1)[0]
When I run the above by hand, or copy-and-paste the file's contents from Atom, it works. However, when I have the Python open the file and read the line itself, it fails:
# This fails, reading from the file from which contents were copy-and-pasted
with open(filename) as curfile:
for line in curfile:
date = line.split('|75=')[1].split('|',1)[0]
This code fails with an IndexError: the split() creates only a single segment, so no [1] segment exists.
When I print the line from the file-based code, it prints smiley faces where the |s should be.
What could be going wrong here? How can I better debug this scenario?
If you're running this from the Windows console (code page 437) there are two vertical bar characters: b'\x7c' and b'\xb3'. The first is part of the ASCII character set, and the second is one of the line-drawing characters that were part of the original PC.
>>> print(b'\x7c\xb3'.decode('cp437'))
|│
In addition you appear to be using a text editor that shows b'\x01' as a vertical bar as well. That's a non-standard way of displaying that character, which is generally invisible since it's an ASCII/Unicode control character.
Once you've figured out the actual character in the file, you can substitute it in your split call.
I'm trying to have Python copy the contents of a .txt file into the bash terminal on OS X (10.10), but the line does not print until every single character of the line has been printed to the line. Is there any way to have Python print each line character-by-character, instead of line-by-line? My code is designed to wait between characters, but each line simply takes a long time to print:
while True:
character = text_file.read(1)
if not character: break
else:
sys.stdout.write(character)
time.sleep(0.050)
When I run this code in IDLE, the characters print one at a time. In Terminal, lines take several seconds to print, and each line prints all at once. Is there any way to reproduce the behavior I'm seeing in IDLE in Terminal?
Add sys.stdout.flush() after sys.stdout.write(character)
The reason should be that the output of stdout is buffered.
if you want remove new line at the end of the line.
you can simply
print character,
will remove the new line(\n).
I am beginning to learn Python and want to use it to automate a process.
The process consists in
modifying a few lines of a file
use the file as the input for an executable
save, move, etc
repeat
The problem is that the file I'm trying to modify was written in a language that utilizes the SUB character to run. Therefore, when I try
with open(myFile,'r') as file:
data = list(file)
data does not contain any information beyond the SUB character.
Therefore, I need to be able to do two things:
Read the whole file in python (without exiting prematurely at the SUB character locations) so that I can modify it.
Be able to run it on the executable (that is, the SUB characters need to be back at their respective places).
Any suggestions on how to go about solving this problem?
Thanks
Use the binary mode to open file.
with open(myFile,'rb') as file:
for line in file:
print line
Are you on Windows? Quoted from your link to the SUB character:
In CP/M, 86-DOS, MS-DOS, PC DOS, DR-DOS and their various derivatives, character 26 was also used to indicate the end of a character stream, and thereby used to terminate user input in an interactive command line window (and as such, often used to finish console input redirection, e.g. as instigated by COPY CON: TYPEDTXT.TXT).
While no longer technically required to indicate the end of a file many text editors and program languages up to the present still support this convention...
Python 2.7 in text mode will stop at a CTRL-Z character (hex 1A), so open the file in binary mode:
Example:
# Create a file with embedded character 1Ah
with open('sub.txt','wb') as f:
f.write(b'abc\x1adef')
# Open in default (text) mode and read as much as possible
with open('sub.txt','r') as f:
print repr(f.read())
# Open in binary mode
with open('sub.txt','rb') as f:
print repr(f.read())
Output:
'abc'
'abc\x1adef'