Develop Reference

About UnicodeDecodeError

I am writing a program to count the words with python(3.6), the code runs smoothly from the terminal. But if I use python IDLE, below error happens:
Traceback (most recent call last):
File "/Users/zhangchaont/python/Course Python Programming/6.7V2.py", line 122, in <module>
main()
File "/Users/zhangchaont/python/Course Python Programming/6.7V2.py", line 21, in main
for line in txtFile:
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 33: ordinal not in range(128)
How to solve this?

Since there is not much info about your code. I can only suggest instead of codecs you can also use this package.
https://github.com/iki/unidecode. The method below should solve your problem. Open your file with open method, and pass it the file_handle.read()
unidecode.unidecode_expect_nonascii(string)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

I have written the code below that provides the path to an Excel file that will be created with the XLWT module
master_path = r"C:\Users\nbt8ye8\Documents\Docs\Report Automation\KBE Reporting\Reports"
master_excel_file_raw = "KBE Master Data.xls"
master_excel_file = os.path.join(master_path, master_excel_file_raw)
Then later in the code I create the Excel file (which works without a problem) with the code below:
master_excel_wbook = xlwt.Workbook()
master_excel_wsheet = master_excel_wbook.add_sheet("All Data", cell_overwrite_ok=True)
master_excel_wbook.save(master_excel_file)
master_excel_wbook.save(tempfile.TemporaryFile())
However when I run the code it gives me the errors below.
Traceback (most recent call last):
File "C:\Users\nbt8ye8\workspace\Report Automation\import_data.py", line 1225, in <module>
createExcelFile()
File "C:\Users\nbt8ye8\workspace\Report Automation\import_data.py", line 1219, in createExcelFile
master_excel_wbook.save(master_excel_file)
File "build\bdist.win32\egg\xlwt\Workbook.py", line 662, in save
File "build\bdist.win32\egg\xlwt\Workbook.py", line 637, in get_biff_data
File "build\bdist.win32\egg\xlwt\Workbook.py", line 599, in __sst_rec
File "build\bdist.win32\egg\xlwt\BIFFRecords.py", line 76, in get_biff_record
File "build\bdist.win32\egg\xlwt\BIFFRecords.py", line 91, in _add_to_sst
File "build\bdist.win32\egg\xlwt\UnicodeUtils.py", line 50, in upack2
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)
Does anyone know how to resolve this issue? I have tried encoding and decoding the strings and so far it has not worked but it is also highly likely I did not do it correctly. Any help would be greatly apprecieated. Thank you!

#RyanG was correct, this error was thrown because there was unicode data in my Excel file. Once I modified the Excel file to remove the unicode data the problem was resolved. Thanks again.

import CGIHTTPServer results UnicodeDecodeError

import CGIHTTPServer results UnicodeDecodeError on Python 2.7.6.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\CGIHTTPServer.py", line 30, in <module>
import SimpleHTTPServer
File "C:\Python27\lib\SimpleHTTPServer.py", line 27, in <module>
class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
File "C:\Python27\lib\SimpleHTTPServer.py", line 208, in SimpleHTTPRequestHand
ler
mimetypes.init() # try to read system mime.types
File "C:\Python27\lib\mimetypes.py", line 358, in init
db.read_windows_registry()
File "C:\Python27\lib\mimetypes.py", line 258, in read_windows_registry
for subkeyname in enum_types(hkcr):
File "C:\Python27\lib\mimetypes.py", line 249, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal
not in range(128)"
Please advise me.

This is a bug in Python 2.7, reported as issue 21138; the mimetypes module doesn't use the correct encoding for Unicode mimetypes in the Windows registry.
The bug ticket contains a work-around; you'll have to edit mimetypes.py.

Python ez_install : UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 11

Getting the error below trying to install ez_install, Windows 7 64 bit machine with fresh Python 2.7. Any ideas?
Installing Setuptools
Traceback (most recent call last):
File "setup.py", line 17, in
exec(init_file.read(), command_ns)
File "", line 8, in
File "c:\users\namar\appdata\local\temp\tmp1tanvy\setuptools-2.1\setuptools\__
init__.py", line 11, in
from setuptools.extension import Extension
File "c:\users\namar\appdata\local\temp\tmp1tanvy\setuptools-2.1\setuptools\ex
tension.py", line 5, in
from setuptools.dist import _get_unpatched
File "c:\users\namar\appdata\local\temp\tmp1tanvy\setuptools-2.1\setuptools\di
st.py", line 15, in
from setuptools.compat import numeric_types, basestring
File "c:\users\namar\appdata\local\temp\tmp1tanvy\setuptools-2.1\setuptools\co
mpat.py", line 19, in
from SimpleHTTPServer import SimpleHTTPRequestHandler
File "c:\python27\lib\SimpleHTTPServer.py", line 27, in
class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
File "c:\python27\lib\SimpleHTTPServer.py", line 208, in SimpleHTTPRequestHand
ler
mimetypes.init() # try to read system mime.types
File "c:\python27\lib\mimetypes.py", line 358, in init
db.read_windows_registry()
File "c:\python27\lib\mimetypes.py", line 258, in read_windows_registry
for subkeyname in enum_types(hkcr):
File "c:\python27\lib\mimetypes.py", line 249, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 11: ordinal
not in range(128)
Something went wrong during the installation.
See the error message above.
C:\Users\namar\Downloads>cd\
C:\>cd Python27

I met the same problem, solved it by deleting non-ASCII character in window registry "HKEY_CLASSES_ROOT\MIME\Database\Content Type" and it works.The reason is
ctype = ctype.encode(default_encoding)
will throw UnicodeDecodeError when it met non-ASCII characters and the program will stop.

Check this link.(http://blog.csdn.net/hugleecool/article/details/17996993).
In C:\Python27\Lib\mimetypes.py find next string
default_encoding = sys.getdefaultencoding()
replace it with
if sys.getdefaultencoding() != 'gbk':
reload(sys)
sys.setdefaultencoding('gbk')
default_encoding = sys.getdefaultencoding()
Notice: instead 'gbk' choose your encoding.

In C:\Python27\Lib\mimetypes.py find next string
default_encoding = sys.getdefaultencoding()
replace it with
if sys.getdefaultencoding() != 'gbk':
reload(sys)
sys.setdefaultencoding('gbk')
default_encoding = sys.getdefaultencoding()
Notice: instead 'gbk' choose your encoding.
The above answer satisfies my case of unicode decode error in using google app engine local web server for php and python .

In my case, simply comment the following four line worked,
try:
ctype = ctype.encode(default_encoding) # omit in 3.x!
except UnicodeEncodeError:
pass
And replace UnicodeEncodeError to be UnicodeError worked.
For more info, you can check this question.

Python rdflib Not Parsing Creative Commons License Information Correctly

I was using rdflib version 3.2.3 and everything was working fine. After upgrading to 4.0.1 I started getting the error:
RDFa parsing Error! 'ascii' codec can't decode byte 0xc3 in position 5454: ordinal not in range(128)
I tried various way to make this work but so far have not succeeded. Below are my attempts.
In each case I:
from rdflib import Graph
First attempt:
>>> lg =Graph()
>>> len(lg.parse('http://creativecommons.org/licenses/by/3.0/'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/graph.py", line 1002, in parse
parser.parse(source, self, **args)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/plugins/parsers/structureddata.py", line 268, in parse
vocab_cache=vocab_cache)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/plugins/parsers/structureddata.py", line 148, in _process
_check_error(processor_graph)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/plugins/parsers/structureddata.py", line 57, in _check_error
raise Exception("RDFa parsing Error! %s" % msg)
Exception: RDFa parsing Error! 'ascii' codec can't decode byte 0xc3 in position 4801: ordinal not in range(128)
Second attempt:
>>> lg =Graph()
>>> len(lg.parse('http://creativecommons.org/licenses/by/3.0/rdf'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/graph.py", line 1002, in parse
parser.parse(source, self, **args)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/plugins/parsers/rdfxml.py", line 570, in parse
self._parser.parse(source)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 349, in end_element_ns
self._cont_handler.endElementNS(pair, None)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/plugins/parsers/rdfxml.py", line 160, in endElementNS
self.current.end(name, qname)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/plugins/parsers/rdfxml.py", line 461, in property_element_end
current.data, literalLang, current.datatype)
File "/home/alex/Projects/RDF/rdfEnv/local/lib/python2.7/site-packages/rdflib/term.py", line 541, in __new__
raise Exception("'%s' is not a valid language tag!"%lang)
Exception: 'i18n' is not a valid language tag!
Third attempt: gives no errors but also does not give any results
>>> lg =Graph()
>>> len(lg.parse('http://creativecommons.org/licenses/by/3.0/rdf', format='rdfa'))
0
So someone please tell me what I am dong wrong! :)

As Graham replied on the rdflib mailinglist, there is a html5lib problem - we will pin it correctly for python 2 for the next release, but for now just do:
pip install html5lib==0.95
The second problem is a problem in the data from creative commons, "i18n" really isn't a valid language tag according to rfc5646. I added the check, but in retrospect it seems to strict to raise an exception. I guess I'll change it to a warning.