Develop Reference

Which encoding should Python open function use?

I'm getting an exception when reading a file that contains a RIGHT DOUBLE QUOTATION MARK Unicode symbol. It is encoded in UTF-8 (0xE2 0x80 0x9D). The minimal example:
import sys
print(sys.getdefaultencoding())
f = open("input.txt", "r")
r.readline()
This script fails reading the first line even if the right quotation mark is not on the first line. The exception looks like that:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python36\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 102: char
acter maps to <undefined>
The input file is in utf-8 encoding, I've tried both with and without BOM. The default encoding returned by sys.getdefaultencoding() is utf-8.
This script fails on the machine with Python 3.6.5 but works well on another with Python 3.6.0. Both machines are Windows.
My questions are mostly theoretical, as this exception is thrown from external software that I cannot change, and it reads file that I don't wish to change. What should be the difference in these machines except the Python patch version? Why does vanilla open use cp1252 if the system default is utf-8?

As clearly stated in Python's open documentation:
In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.
Windows defaults to a localized encoding (cp1252 on US and Western European versions). Linux typically defaults to utf-8.
Because it is platform-dependent, use the encoding parameter and specify the encoding of the file explicitly.

Why did I get UnicodeDecodeError when I read a file which contains Chinese characters?

>>> path = 'name.txt'
>>> content = None
>>> with open(path, 'r') as file:
... content = file.readlines()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/mnt/lustre/share/miniconda3/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 163: ordinal not in range(128)
When I run this code to read a file which contains Chinese characters, I got an error. The file is saved by using UTF-8. My python version is 3.6.5. But it runs ok in python2.7.

open is using the ASCII codec to try to read the file. The easiest way to fix this is to specify the encoding:
with open(path, 'r', encoding='utf-8') as file:
Your locale should probably specify the preferred encoding as UTF-8, but I think it depends on OS and language settings.

Python 2.7 reads files into byte strings by default.
Python 3.x reads files into Unicode strings by default, so the bytes in the file must be decoded.
The default encoding used varies by operating system, but can be determined by calling locale.getpreferredencoding(False). This is often utf8 on Linux systems, but Windows systems return a localized ANSI encoding, e.g. cp1252 for US/Western European Windows versions.
In Python 3, specify the encoding you expect for files so as not to rely on a locale-specific default. For example:
with open(path,'r',encoding='utf8') as f:
...
You can do this in Python 2 as well, but use io.open(), which is compatible with Python 3's open() and will read Unicode strings instead of byte strings. io.open() is available in Python 3 as well for portability.

what encoding should I use to open a file with a big letter N with tilde character?

I'm trying to open a file with a big letter N tilde (http://graphemica.com/%C3%91) but I can't seem to figure it out. when I open the file in notepad++ it shows the character as xD1, when I open the file in gedit it shows \D1. When I open the file in excel, it shows the character correctly.
Now I'm trying to open the file in python, it halts when it encounters the character. I'm aware that I can put in the encoding so the file can be opened properly but I'm not sure which encoding I should use. My error is
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 0: invalid continuation byte
this is my code
with codecs.open('tsv.txt', 'r', 'utf8') as my_file:
for line in my_file:
print(line)
if it is not utf8, then what should I use? From the site above, it does not show which encoding 0xd1 is associated with.

You can find in tables how 'Ñ' gets encoded in different encodings.
You can also try it directly with Python:
>>> 'Ñ'.encode('utf8')
b'\xc3\x91'
>>> 'Ñ'.encode('latin1')
b'\xd1'
It seems that your file is encoded in latin-1.

UnicodeDecodeError even when importing simple txt file in Python

I would like to import even a simple text file into Python. For example, here's the contents of example.txt:
hello
my
friend
Very simple. However, when I try to import the file and read it:
f = open('example.txt')
f.read()
I get the following error:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
f.read()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
What's the source of this problem? Clearly there are not any non-ascii characters in the file.
I've tried this in IDLE, terminal (Mac OSX) and Rodeo and get similar issues in all.
I'm very new to Python and am concerned I may have screwed up something in installation. I've downloaded various versions over the years, straight from Python, Anaconda, macports, etc. and I'm wondering if the various sources are not playing nicely...
Python 3.5.1 on OSX 10.11.4.

Maybe your file is saved with the encoding UTF-8 with BOM (Byte order mark). Try to save you file explicit as UTF-8 (without BOM). While the BOM is not included in the ASCII codec, it causes an UnicodeError.
Hope this helps!

Changing the preferred encoding for Windows7 command prompt

Using Python3.3
Running a python script to do some CAD file conversion, but getting below error
Conversion Failed
Traceback (most recent call last):
File "Start.py", line 141, in convertLib
lib.writeLibrary(modFile,symFile)
File "C:\Python33\Eagle2Kicad/Library\Library.py", line 67, in writeLibrary
self.writeSymFile(symFile)
File "C:\Python33\Eagle2Kicad/Library\Library.py", line 88, in writeSymFile
devicepart.write(symFile)
File "C:\Python33\Eagle2Kicad/Common\Symbol.py", line 51, in write
symbol.write(symFile)
File "C:\Python33\Eagle2Kicad/Common\Symbol.py", line 114, in write
symFile.write(pin.symRep())
File "C:\Python33\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\x96' in position 4: character maps to <undefined>
It seems the preferred encoding in my Windows7 command prompt is NOT cp1252 (typing chcp shows "Active code page 437"). How can I change it to cp1252?
Can anyone please suggest?
EDIT:
As the default preferred encoding of the python command line is cp1252, I tried running the script from python command line instead of the windows command prompt. But I am still getting the same error as above. Can anyone please suggest?

When writing to files, the console encoding is not taken into account; only the locale is. From the open() function documenation:
In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.
On your system, evidently that call returns 'cp1252'. The remedy is to always name the encoding for files explicitly; pass in an encoding argument to set the desired encoding for files:
with open(filename, 'w', encoding='utf8') as symFile: