Develop Reference

How to handle Encoding Errors in Python [duplicate]

when I try to concatenate this, I get the UnicodeDecodeError when the field contains 'ñ' or '´'. If the field that contains the 'ñ' or '´' is the last I get no error.
#...
nombre = fabrica
nombre = nombre.encode("utf-8") + '-' + sector.encode("utf-8")
nombre = nombre.encode("utf-8") + '-' + unidad.encode("utf-8")
#...
return nombre
any idea? Many thanks!

You are encoding to UTF-8, then re-encoding to UTF-8. Python can only do this if it first decodes again to Unicode, but it has to use the default ASCII codec:
>>> u'ñ'
u'\xf1'
>>> u'ñ'.encode('utf8')
'\xc3\xb1'
>>> u'ñ'.encode('utf8').encode('utf8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
Don't keep encoding; leave encoding to UTF-8 to the last possible moment instead. Concatenate Unicode values instead.
You can use str.join() (or, rather, unicode.join()) here to concatenate the three values with dashes in between:
nombre = u'-'.join(fabrica, sector, unidad)
return nombre.encode('utf-8')
but even encoding here might be too early.
Rule of thumb: decode the moment you receive the value (if not Unicode values supplied by an API already), encode only when you have to (if the destination API does not handle Unicode values directly).

When you get a UnicodeEncodeError, it means that somewhere in your code you convert directly a byte string to a unicode one. By default in Python 2 it uses ascii encoding, and utf8 encoding in Python3 (both may fail because not every byte is valid in either encoding)
To avoid that, you must use explicit decoding.
If you may have 2 different encoding in your input file, one of them accepts any byte (say UTF8 and Latin1), you can try to first convert a string with first and use the second one if a UnicodeDecodeError occurs.
def robust_decode(bs):
'''Takes a byte string as param and convert it into a unicode one.
First tries UTF8, and fallback to Latin1 if it fails'''
cr = None
try:
cr = bs.decode('utf8')
except UnicodeDecodeError:
cr = bs.decode('latin1')
return cr
If you do not know original encoding and do not care for non ascii character, you can set the optional errors parameter of the decode method to replace. Any offending byte will be replaced (from the standard library documentation):
Replace with a suitable replacement character; Python will use the official U+FFFD REPLACEMENT CHARACTER for the built-in Unicode codecs on decoding and ‘?’ on encoding.
bs.decode(errors='replace')

I was getting this error when executing in python3,I got the same program working by simply executing in python2

Python: increment special character Í

I want to read some words from an excel file and extracte some information.
Reading the file is no problem.
The point is, that I want to increment the last character of a word. It is no problem for normal characters like 'A'. But special Characters like 'Í' are a problem.
I read the content with this:
val = val.encode('utf-8')
I put this value in a dictionary.
The next step is to iterate through the dict and get the saved information. info['streettype'] contains my val from before. Now i convert the value to upper case like this:
w2 = info['streettype'].decode('utf-8').upper().encode('utf-8')
That is needed because some characters are special, like I said (e.g. 'é', 'ž', 'í').
Now I want to increment the last character of the word, which can be a special character.
w3 = w2.decode('utf-8')[:-1].encode('utf-8')
lastLetter = w2.decode('utf-8')[-1].encode('utf-8')
Now I increment the character by using:
lastLetter2 = (chr(ord(lastLetter.decode('utf-8')) + 1))
Next I want to save it in a text file.
I want to save the original word and the edited word.
I think I need to reencode my lastLetter2, but it does not work.
When I just save my w2 and w3+lastLetter2 I get strange results because some are encoded, some are not.
For the word:
NÁBŘEŽÍ
my Result is:
"NÃBŘEŽÃ", "NÃBŘEŽÎÃ"
but I want:
"NÁBŘEŽÍ", "NÁBŘEŽÎ"
(Í is ascii 205, Î is ascii 206)
Can someone help me to save this problem?

Stop encoding your data to UTF-8 all the time; keep your text as Unicode, it makes processing much easier. Leave encoding to the last minute, preferably by having the file object encode this for you.
Having the file encode Unicode means that in Python 2 you'd use io.open() rather than the standard built-in open() function; this is the same infrastructure Python 3 uses to handle Unicode and file I/O.
You managed to create a Mojibake by encoding and decoding at will here; your text is now a mix of UTF-8 data decoded with Windows codepage 1252 then encoded to UTF8 again, plus non-mangled data:
>>> print u'NÃBŘEŽÃ'
NÃBŘEŽÃ
>>> print u'NÃBŘEŽÃ'[3:-1].encode('cp1252').decode('utf8')
ŘEŽ
Note that the last character in the first stringis invalid; it is missing a byte! That's because the result of 'decoding' the last character's UTF-8 bytes should not have been possible in a proper CP1252 codec; I had to use the ftfy project internal repair codecs to bypass that problem:
>>> print u'NÃBŘEŽÃ\x8d'[3:].encode('sloppy-cp1252').decode('utf8')
ŘEŽÍ
>>> u'Í'.encode('utf8').decode('cp1252')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/cp1252.py", line 15, in decode
return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 1: character maps to <undefined>
>>> u'Í'.encode('utf8').decode('sloppy-cp1252')
u'\xc3\x8d'
>>> print u'Í'.encode('utf8').decode('sloppy-cp1252')
Ã
The only way to fix this is to a) ensure you read your data using the correct codecs, and b) then treat all text as Unicode throughout your code, and only encode at the last moment to the correct output codec.
Handling Unicode code points with ord() and unichr() (in Python 2) and chr() in Python 3 will then work as expected:
>>> lastletter = u'Î'
>>> ord(lastletter)
206
>>> unichr(ord(lastletter) + 1)
u'\xcf'
>>> print unichr(ord(lastletter) + 1)
Ï
You may want to read up on Python and Unicode:
Pragmatic Unicode by Ned Batchelder
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky
The Python Unicode HOWTO

Python, Encoding output to UTF-8

I have a definition that builds a string composed of UTF-8 encoded characters. The output files are opened using 'w+', "utf-8" arguments.
However, when I try to x.write(string) I get the UnicodeEncodeError: 'ascii' codec can't encode character u'\ufeff' in position 1: ordinal not in range(128)
I assume this is because normally for example you would do `print(u'something'). But I need to use a variable and the quotations in u'_' negate that...
Any suggestions?
EDIT: Actual code here:
source = codecs.open("actionbreak/" + target + '.csv','r', "utf-8")
outTarget = codecs.open("actionbreak/" + newTarget, 'w+', "utf-8")
x = str(actionT(splitList[0], splitList[1]))
outTarget.write(x)
Essentially all this is supposed to be doing is building me a large amount of strings that look similar to this:
[日木曜 Deliverables]= CASE WHEN things = 11
THEN C ELSE 0 END

Are you using codecs.open()? Python 2.7's built-in open() does not support a specific encoding, meaning you have to manually encode non-ascii strings (as others have noted), but codecs.open() does support that and would probably be easier to drop in than manually encoding all the strings.
As you are actually using codecs.open(), going by your added code, and after a bit of looking things up myself, I suggest attempting to open the input and/or output file with encoding "utf-8-sig", which will automatically handle the BOM for UTF-8 (see http://docs.python.org/2/library/codecs.html#encodings-and-unicode, near the bottom of the section) I would think that would only matter for the input file, but if none of those combinations (utf-8-sig/utf-8, utf-8/utf-8-sig, utf-8-sig/utf-8-sig) work, then I believe the most likely situation would be that your input file is encoded in a different Unicode format with BOM, as Python's default UTF-8 codec interprets BOMs as regular characters so the input would not have an issue but output could.
Just noticed this, but... when you use codecs.open(), it expects a Unicode string, not an encoded one; try x = unicode(actionT(splitList[0], splitList[1])).
Your error can also occur when attempting to decode a unicode string (see http://wiki.python.org/moin/UnicodeEncodeError), but I don't think that should be happening unless actionT() or your list-splitting does something to the Unicode strings that causes them to be treated as non-Unicode strings.

In python 2.x there are two types of string: byte string and unicode string. First one contains bytes and last one - unicode code points. It is easy to determine, what type of string it is - unicode string starts with u:
# byte string
>>> 'abc'
'abc'
# unicode string:
>>> u'abc абв'
u'abc \u0430\u0431\u0432'
'abc' chars are the same, because the are in ASCII range. \u0430 is a unicode code point, it is out of ASCII range. "Code point" is python internal representation of unicode points, they can't be saved to file. It is needed to encode them to bytes first. Here how encoded unicode string looks like (as it is encoded, it becomes a byte string):
>>> s = u'abc абв'
>>> s.encode('utf8')
'abc \xd0\xb0\xd0\xb1\xd0\xb2'
This encoded string now can be written to file:
>>> s = u'abc абв'
>>> with open('text.txt', 'w+') as f:
... f.write(s.encode('utf8'))
Now, it is important to remember, what encoding we used when writing to file. Because to be able to read the data, we need to decode the content. Here what data looks like without decoding:
>>> with open('text.txt', 'r') as f:
... content = f.read()
>>> content
'abc \xd0\xb0\xd0\xb1\xd0\xb2'
You see, we've got encoded bytes, exactly the same as in s.encode('utf8'). To decode it is needed to provide coding name:
>>> content.decode('utf8')
u'abc \u0430\u0431\u0432'
After decode, we've got back our unicode string with unicode code points.
>>> print content.decode('utf8')
abc абв

xgord is right, but for further edification it's worth noting exactly what \ufeff means. It's known as a BOM or a byte order mark and basically it's a callback to the early days of unicode when people couldn't agree which way they wanted their unicode to go. Now all unicode documents are prefaced with either an \ufeff or an \uffef depending on which order they decide to arrange their bytes in.
If you hit an error on those characters in the first location you can be sure the issue is that you are not trying to decode it as utf-8, and the file is probably still fine.

string encoding and decoding?

Here are my attempts with error messages. What am I doing wrong?
string.decode("ascii", "ignore")
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in
position 37: ordinal not in range(128)
string.encode('utf-8', "ignore")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
37: ordinal not in range(128)

You can't decode a unicode, and you can't encode a str. Try doing it the other way around.

Guessing at all the things omitted from the original question, but, assuming Python 2.x the key is to read the error messages carefully: in particular where you call 'encode' but the message says 'decode' and vice versa, but also the types of the values included in the messages.
In the first example string is of type unicode and you attempted to decode it which is an operation converting a byte string to unicode. Python helpfully attempted to convert the unicode value to str using the default 'ascii' encoding but since your string contained a non-ascii character you got the error which says that Python was unable to encode a unicode value. Here's an example which shows the type of the input string:
>>> u"\xa0".decode("ascii", "ignore")
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
u"\xa0".decode("ascii", "ignore")
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 0: ordinal not in range(128)
In the second case you do the reverse attempting to encode a byte string. Encoding is an operation that converts unicode to a byte string so Python helpfully attempts to convert your byte string to unicode first and, since you didn't give it an ascii string the default ascii decoder fails:
>>> "\xc2".encode("ascii", "ignore")
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
"\xc2".encode("ascii", "ignore")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

Aside from getting decode and encode backwards, I think part of the answer here is actually don't use the ascii encoding. It's probably not what you want.
To begin with, think of str like you would a plain text file. It's just a bunch of bytes with no encoding actually attached to it. How it's interpreted is up to whatever piece of code is reading it. If you don't know what this paragraph is talking about, go read Joel's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets right now before you go any further.
Naturally, we're all aware of the mess that created. The answer is to, at least within memory, have a standard encoding for all strings. That's where unicode comes in. I'm having trouble tracking down exactly what encoding Python uses internally for sure, but it doesn't really matter just for this. The point is that you know it's a sequence of bytes that are interpreted a certain way. So you only need to think about the characters themselves, and not the bytes.
The problem is that in practice, you run into both. Some libraries give you a str, and some expect a str. Certainly that makes sense whenever you're streaming a series of bytes (such as to or from disk or over a web request). So you need to be able to translate back and forth.
Enter codecs: it's the translation library between these two data types. You use encode to generate a sequence of bytes (str) from a text string (unicode), and you use decode to get a text string (unicode) from a sequence of bytes (str).
For example:
>>> s = "I look like a string, but I'm actually a sequence of bytes. \xe2\x9d\xa4"
>>> codecs.decode(s, 'utf-8')
u"I look like a string, but I'm actually a sequence of bytes. \u2764"
What happened here? I gave Python a sequence of bytes, and then I told it, "Give me the unicode version of this, given that this sequence of bytes is in 'utf-8'." It did as I asked, and those bytes (a heart character) are now treated as a whole, represented by their Unicode codepoint.
Let's go the other way around:
>>> u = u"I'm a string! Really! \u2764"
>>> codecs.encode(u, 'utf-8')
"I'm a string! Really! \xe2\x9d\xa4"
I gave Python a Unicode string, and I asked it to translate the string into a sequence of bytes using the 'utf-8' encoding. So it did, and now the heart is just a bunch of bytes it can't print as ASCII; so it shows me the hexadecimal instead.
We can work with other encodings, too, of course:
>>> s = "I have a section \xa7"
>>> codecs.decode(s, 'latin1')
u'I have a section \xa7'
>>> codecs.decode(s, 'latin1')[-1] == u'\u00A7'
True
>>> u = u"I have a section \u00a7"
>>> u
u'I have a section \xa7'
>>> codecs.encode(u, 'latin1')
'I have a section \xa7'
('\xa7' is the section character, in both
Unicode and Latin-1.)
So for your question, you first need to figure out what encoding your str is in.
Did it come from a file? From a web request? From your database? Then the source determines the encoding. Find out the encoding of the source and use that to translate it into a unicode.
s = [get from external source]
u = codecs.decode(s, 'utf-8') # Replace utf-8 with the actual input encoding
Or maybe you're trying to write it out somewhere. What encoding does the destination expect? Use that to translate it into a str. UTF-8 is a good choice for plain text documents; most things can read it.
u = u'My string'
s = codecs.encode(u, 'utf-8') # Replace utf-8 with the actual output encoding
[Write s out somewhere]
Are you just translating back and forth in memory for interoperability or something? Then just pick an encoding and stick with it; 'utf-8' is probably the best choice for that:
u = u'My string'
s = codecs.encode(u, 'utf-8')
newu = codecs.decode(s, 'utf-8')
In modern programming, you probably never want to use the 'ascii' encoding for any of this. It's an extremely small subset of all possible characters, and no system I know of uses it by default or anything.
Python 3 does its best to make this immensely clearer simply by changing the names. In Python 3, str was replaced with bytes, and unicode was replaced with str.

That's because your input string can’t be converted according to the encoding rules (strict by default).
I don't know, but I always encoded using directly unicode() constructor, at least that's the ways at the official documentation:
unicode(your_str, errors="ignore")

List of unicode strings

If I have a list of unicode strings
lst = [ u"aaa", u"bbb", u"foo", u"bar", ... u"baz", u"zzz" ]
is it necessary to write a prefix u before every string? Can I make a construction that says that every element of lst will be unicode string and then write it without u prefix?

In Python 2.7 (also Python 2.6) you can make unicode literals the default for a module:
from __future__ import unicode_literals
You must include the import at the top of the file, and it then applies to all string literals in the file. Use a b prefix to force byte strings:
>>> from __future__ import unicode_literals
>>> "sss"
u'sss'
>>> b"x"
'x'

If your intention is to convert a set of standard strings to unicode, you could map that function onto your list:
lst = ["aaa", "bbb", "ccc"]
map(unicode, lst)
Which gives
[u"aaa", u"bbb", u"ccc"]
If however lst contains a non ASCII character string, you'll have to prefix that particular string with the u. If you don't, you'll get this error on the conversion:
lst = ["\xe4"]
map(unicode,lst)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
As noted in the comments, this answer is different for Python 2.x or 3.x. In Python 3, everything changes:
Everything you thought you knew about binary data and Unicode has changed. Python 3.0 uses the concepts of text and (binary) data instead of Unicode strings and 8-bit strings. All text is Unicode; however encoded Unicode is represented as binary data. The type used to hold text is str, the type used to hold data is bytes. The biggest difference with the 2.x situation is that any attempt to mix text and data in Python 3.0 raises TypeError, whereas if you were to mix Unicode and 8-bit strings in Python 2.x, it would work if the 8-bit string happened to contain only 7-bit (ASCII) bytes, but you would get UnicodeDecodeError if it contained non-ASCII values. This value-specific behavior has caused numerous sad faces over the years.