Develop Reference

Which encoding should Python open function use?

I'm getting an exception when reading a file that contains a RIGHT DOUBLE QUOTATION MARK Unicode symbol. It is encoded in UTF-8 (0xE2 0x80 0x9D). The minimal example:
import sys
print(sys.getdefaultencoding())
f = open("input.txt", "r")
r.readline()
This script fails reading the first line even if the right quotation mark is not on the first line. The exception looks like that:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python36\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 102: char
acter maps to <undefined>
The input file is in utf-8 encoding, I've tried both with and without BOM. The default encoding returned by sys.getdefaultencoding() is utf-8.
This script fails on the machine with Python 3.6.5 but works well on another with Python 3.6.0. Both machines are Windows.
My questions are mostly theoretical, as this exception is thrown from external software that I cannot change, and it reads file that I don't wish to change. What should be the difference in these machines except the Python patch version? Why does vanilla open use cp1252 if the system default is utf-8?

As clearly stated in Python's open documentation:
In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.
Windows defaults to a localized encoding (cp1252 on US and Western European versions). Linux typically defaults to utf-8.
Because it is platform-dependent, use the encoding parameter and specify the encoding of the file explicitly.

UnicodeError when importing python file

I am trying to import a python file Sonderbuch_BASECASE_3ph.py into another python file test.py. test.py is in the main dir foo while Sonderbuch_BASECASE_3ph.py is in a subdir grid_data.
Sonderbuch_BASECASE_3ph.py has a function with the same name, which I need to import as well:
# Sonderbuch_BASECASE_3ph
from numpy import array
def Sonderbuch_BASECASE_3ph():
.....
Both of these attempts to import result in a SyntaxError:
from grid_data import Sonderbuch_BASECASE_3ph
import grid_data.Sonderbuch_BASECASE_3ph
Output:
Traceback (most recent call last):
File "C:/Users/Artur/Desktop/foo/test.py", line 1, in <module>
from grid_data import Sonderbuch_BASECASE_3ph
File "C:\Users\Artur\Desktop\foo\grid_data\Sonderbuch_BASECASE_3ph.py", line 1550
SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xe4 in position 29: invalid continuation byte
Edit:
The encoding of the file seems to be windows-1252, at least that is what pycharm is proposing. Decoding the file in windows-1252 does not solve the ErrorMsg though. Sonderbuch_BASECASE_3hp.py is just a storage file for a dictionary. I was hoping I could just import it.
None of the encodings seem to work.

What's in your Sonderbuch_BASECASE_3ph.py file exactly?
I guess that the files use different encoding hence importing one to another may result in error. My guess is that your test.py is in UTF-8 while the other file is encoded with latin-1 or something like that. Check what's the encoding of the files (you can do it in PyCharm, Sublime, Notepad++, etc.). In Pycharm, you can see the encoding of a file at the bottom right (by default).

Why did I get UnicodeDecodeError when I read a file which contains Chinese characters?

>>> path = 'name.txt'
>>> content = None
>>> with open(path, 'r') as file:
... content = file.readlines()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/mnt/lustre/share/miniconda3/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 163: ordinal not in range(128)
When I run this code to read a file which contains Chinese characters, I got an error. The file is saved by using UTF-8. My python version is 3.6.5. But it runs ok in python2.7.

open is using the ASCII codec to try to read the file. The easiest way to fix this is to specify the encoding:
with open(path, 'r', encoding='utf-8') as file:
Your locale should probably specify the preferred encoding as UTF-8, but I think it depends on OS and language settings.

Python 2.7 reads files into byte strings by default.
Python 3.x reads files into Unicode strings by default, so the bytes in the file must be decoded.
The default encoding used varies by operating system, but can be determined by calling locale.getpreferredencoding(False). This is often utf8 on Linux systems, but Windows systems return a localized ANSI encoding, e.g. cp1252 for US/Western European Windows versions.
In Python 3, specify the encoding you expect for files so as not to rely on a locale-specific default. For example:
with open(path,'r',encoding='utf8') as f:
...
You can do this in Python 2 as well, but use io.open(), which is compatible with Python 3's open() and will read Unicode strings instead of byte strings. io.open() is available in Python 3 as well for portability.

Error: "'ascii' codec can't decode byte 0xd8 in position" in Python 3

I wrote a program that recursively searches for files with a particular extension in folders and does some processing. Strangely, the program runs fine with somewhere around 85 files and then crashes on the same file every time. I don't think there's anything different about that file or filename. Because it runs fine for 85 files, I know the error is not about my code per se, but more about the wrong compiler maybe?
OS: Linux arctic 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u1 x86_64 GNU/Linux
Error details (complete Traceback):
Traceback (most recent call last):
File "scoretotal.py", line 98, in <module>
main()
File "scoretotal.py", line 96, in main
find_score_files()
File "scoretotal.py", line 89, in find_score_files
total = calculate_total((os.path.join(root,filename)))
File "scoretotal.py", line 14, in calculate_total
lines = file_object_read.read()
File "/soft/linux/bin/../python3.3.3/lib/python3.3/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position 17: ordinal not in range(128)
I'm running Python 3.3.3. From my research online, it may have something to do with unicode or UTF-8 format, but for the life of me I can't figure it out. What is going wrong?

When you open a file without specifying an encoding, Python will pick one for you; in your case it picked ascii, which is reasonably safe in that it's unlikely to give you back the wrong characters, but runs into errors quite easily. You need to check with the source of those files to find out their encoding and include it in the open call. For example if you've determined that the files were written with ISO-8859-1 encoding:
file_object_read = open(path, 'r', encoding='iso-8859-1')
If you don't know what encoding to use you'll have to guess, and accept that sometimes your guess will be wrong. On Linux you could try 'utf-8' and on Windows you could try 'mbcs', as those are the defaults used by other programs on those systems. There are utilities available that will inspect the file contents and try to make an educated guess, including the chardet package.

Are you familiar with the difference between ' and ‘ or " and “... These (what I call) pretty quotes tend to ruin pythons day.
It may not be this exactly but it is certainly something to do with the character set in the file. As seen here: http://effbot.org/pyfaq/what-does-unicodeerror-ascii-decoding-encoding-error-ordinal-not-in-range-128-mean.htm
I would recommend a try catch, excepting errors and casting to ascii.

My python can only open the saved text file by notepad, why?

I am using Pyhton3.4.1 and win7. I am trying to reading a txt file exported from a software. it seems that python cannot read this text file. But I found if I open the text file by notepad and add a space in any place and save it, the python works well then.
I tried the same code and same file on my mac, it has the same problem as in windows.
For original text file, not working,open and saved in windows notepad, working,
open ans saved in mac textedit, not working.
I am doubting the original coding of the text file might not be right.
Thanks
Python code
InputFileName=input("Please tell me the input file name:")
#StartLNum=int(input("Please tell me the start line number:"))
#EndLNum=int(input("Please tell me the end line number:"))
StartLNum=18
EndLNum=129
lnum=1
OutputName='out'+InputFileName
fw=open(OutputName,'w')
with open(InputFileName,"r") as fr:
for line in fr:
if (lnum >= StartLNum) & (lnum<=EndLNum):
#print(line)
fw.write(line)
lnum+=1
fw.close()
Shell
>>> ================================ RESTART ================================
>>>
Please tell me the input file name:Jul-18-2014.txt
Traceback (most recent call last):
File "C:\Users\Jeremy\Desktop\read.py", line 13, in <module>
for line in fr:
UnicodeDecodeError: 'gbk' codec can't decode byte 0xb3 in position 4309: illegal multibyte sequence
>>> ================================ RESTART ================================
>>>
Please tell me the input file name:Jul-18-2014.txt
>>>
Plus, the error below is the same code reported on my mac(Python3.4.1,OS10.9)
Traceback (most recent call last):
File "/Users/Jeremy/Desktop/read.py", line 14, in <module>
for line in fr:
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb3 in position 4174: ordinal not in range(128)

When you save the file in Notepad, the file is reencoded to be saved as your default file encoding for your Windows installation. Notepad auto-detected the encoding when it opened the file, however.
Python opens file using that same system encoding, by default, which is why you can now open the file. Quoting the open() function documentation:
encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any encoding supported by Python can be used.
You'll have to explicitly specify the correct encoding for the file if you wanted to open it directly in Python:
with open(InputFileName, "r", encoding='utf-8-sig') as fr:
I used 'utf-8-sig' as an example here, as that is a file encoding that Notepad can auto-detect. It could well be that the encoding is UTF-16 or plain UTF-8 or any number of other encodings, however.
If you think that the page is encoded with a specific ANSI codepage you still have to name the exact codepage. Your system is configured to use code page 936 (GBK) but that is not the correct encoding for this file.
See the codecs module for a list of supported encodings.