Hi can you help me decode this message and what to do:
main.py", line 1278, in post
message.body = "%s %s/%s/%s" % (msg, host, ad.key().id(), slugify(ad.title.encode('utf-8')))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
Thanks
UPDATE having tried removing the encode call it appears to work:
class Recommend(webapp.RequestHandler):
def post(self, key):
ad= db.get(db.Key(key))
email = self.request.POST['tip_email']
host = os.environ.get("HTTP_HOST", os.environ["SERVER_NAME"])
senderemail = users.get_current_user().email() if users.get_current_user() else 'info#monton.cl' if host.endswith('.cl') else 'info#monton.com.mx' if host.endswith('.mx') else 'info#montao.com.br' if host.endswith('.br') else 'admin#koolbusiness.com'
message = mail.EmailMessage(sender=senderemail, subject="%s recommends %s" % (self.request.POST['tip_name'], ad.title) )
message.to = email
message.body = "%s %s/%s/%s" % (self.request.POST['tip_msg'],host,ad.key().id(),slugify(ad.title))
message.send()
matched_images=ad.matched_images
count = matched_images.count()
if ad.text:
p = re.compile(r'(www[^ ]*|http://[^ ]*)')
text = p.sub(r'\1',ad.text.replace('http://',''))
else:
text = None
self.response.out.write("Message sent<br>")
path = os.path.join(os.path.dirname(__file__), 'market', 'market_ad_detail.html')
self.response.out.write(template.render(path, {'user_url':users.create_logout_url(self.request.uri) if users.get_current_user() else users.create_login_url(self.request.uri),
'user':users.get_current_user(), 'ad.user':ad.user,'count':count, 'ad':ad, 'matched_images': matched_images,}))
The problem here is your underlying model (message.body) only wants ASCII text but you're trying to give it a string encoded in unicode.
But since you've got a normal ascii string here, you can just make python print out the '?' character when you've got a non-ascii-printing string.
"UNICODE STRING".encode('ascii','replace').decode('ascii')
So like from your example above:
message.body = "%s %s/%s/%s" % \
(msgencode('ascii','replace').decode('ascii'),
hostencode('ascii','replace').decode('ascii'),
ad.key().id()encode('ascii','replace').decode('ascii'),
slugify(ad.title)encode('ascii','replace').decode('ascii'))
Or just encode/decode on the variable that has the unicode character.
But this isn't an optimal solution. The best idea is to make message.body a unicode string. Being that doesn't seem feasible (I'm not familiar with GAE), you can use this to at least not have errors.
You've got a Unicode character in a place that you're not supposed to. Most often I find this error is having MS Word-style slanted quotes.
One of these fields has some characters that cannot be encoded. If you switch to python 3 (it has better unicode support), or you change the encoding of the entire script the problem should stop, about the best way to change the encoding in 2.x is using the encoding comment line. If you see http://evanjones.ca/python-utf8.html you will see more of an explanation of using python with utf-8 support the best suggestion is add # -*- coding: utf-8 -*- to the top of your script. And handle scripts like this
s = "hello normal string"
u = unicode( s, "utf-8" )
backToBytes = u.encode( "utf-8" )
I had a similar problem when using Django norel and Google App Engine.
The problem was at the folder containing the application. Probably isn't this the problem described in this question, but, maybe helps someone don't waste time like me.
Try first change you application folder maybe to /home/ and try to run again, if doesn't works, try something more.
Related
I am running into this error when I use options(FromCache()) with Sqlalchemy running on python3.6.5, dogpile.cache==0.7.1 and SQLAlchemy==1.3.2
UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xae’ in position 744: ordinal not in range(128)
I figured out it's because of this the trademark in "BrandX®".
Example:
vendors = ['BrandX®', 'BrandY Inc.']
engine = create_engine(os.getenv('DEV_DATABASE_URL'), client_encoding='utf-8')
Session = scoped_session(sessionmaker(bind=engine, autoflush=False))
store_id = 123
db = Session()
q = db2.query(Order).join(Product) \
.options(FromCache()) \
.filter(Order.store_id == store_id) \
if vendor:
clauses = []
for v in vendor:
clauses.append((Product.vendor == v))
q = q.filter(or_(*clauses))
return q.all()
I tried to change the vendor encoding to 'utf-8' and 'ascii' and it's not working. Appreciate any help.
Ok, after playing around with encoding to no avail, I figured out the error is actually due to the caching. Specifically, the .options(FromCache()) is causing the problem.
I traced the error to a function called md5_key_mangler, and here's the function.
def md5_key_mangler(key):
"""Receive cache keys as long concatenated strings;
distill them into an md5 hash.
"""
return md5(key.encode("ascii")).hexdigest()
Full documentation from Sqlalchemy around dogpile caching.
It appears to be this line
md5(key.encode("ascii")).hexdigest()
that is causing the problem.
I was then able to go into the file containing my dogpile_caching.environment which I got from the attached link and change the key.encode to utf-8.
md5(key.encode("utf-8")).hexdigest()
And that solved the error. Hope that helps!
I'm reading a config file in python getting sections and creating new config files for each section.
However.. I'm getting a decode error because one of the strings contains Español=spain
self.output_file.write( what.replace( " = ", "=", 1 ) )
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)
How would I adjust my code to allow for encoded characters such as these? I'm very new to this so please excuse me if this is something simple..
class EqualsSpaceRemover:
output_file = None
def __init__( self, new_output_file ):
self.output_file = new_output_file
def write( self, what ):
self.output_file.write( what.replace( " = ", "=", 1 ) )
def get_sections():
configFilePath = 'C:\\test.ini'
config = ConfigParser.ConfigParser()
config.optionxform = str
config.read(configFilePath)
for section in config.sections():
configdata = {k:v for k,v in config.items(section)}
confignew = ConfigParser.ConfigParser()
cfgfile = open("C:\\" + section + ".ini", 'w')
confignew.add_section(section)
for x in configdata.items():
confignew.set(section,x[0],x[1])
confignew.write( EqualsSpaceRemover( cfgfile ) )
cfgfile.close()
If you use python2 with from __future__ import unicode_literals then every string literal you write is an unicode literal, as if you would prefix every literal with u"...", unless you explicitly write b"...".
This explains why you get an UnicodeDecodeError on this line:
what.replace(" = ", "=", 1)
because what you actually do is
what.replace(u" = ",u"=",1 )
ConfigParser uses plain old str for its items when it reads a file using the parser.read() method, which means what will be a str. If you use unicode as arguments to str.replace(), then the string is converted (decoded) to unicode, the replacement applied and the result returned as unicode. But if what contains characters that can't be decoded to unicode using the default encoding, then you get an UnicodeDecodeError where you wouldn't expect one.
So to make this work you can
use explicit prefixes for byte strings: what.replace(b" = ", b"=", 1)
or remove the unicode_litreals future import.
Generally you shouldn't mix unicode and str (python3 fixes this by making it an error in almost any case). You should be aware that from __future__ import unicode_literals changes every non prefixed literal to unicode and doesn't automatically change your code to work with unicode in all case. Quite the opposite in many cases.
I have the following function to create cipher text and then save it:
def create_credential(self):
des = DES.new(CIPHER_N, DES.MODE_ECB)
text = str(uuid.uuid4()).replace('-','')[:16]
cipher_text = des.encrypt(text)
return cipher_text
def decrypt_credential(self, text):
des = DES.new(CIPHER_N, DES.MODE_ECB)
return des.decrypt(text)
def update_access_credentials(self):
self.access_key = self.create_credential()
print repr(self.access_key) # "\xf9\xad\xfbO\xc1lJ'\xb3\xda\x7f\x84\x10\xbbv&"
self.access_password = self.create_credential()
self.save()
And I will call:
>>> from main.models import *
>>> u=User.objects.all()[0]
>>> u.update_access_credentials()
And this is the stacktrace I get:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf5 in position 738: invalid start byte
Why is this occurring and how would I get around it?
You are storing a bytestring into a Unicode database field, so it'll try and decode to Unicode.
Either use a database field that can store opaque binary data, decode explicitly to Unicode (latin-1 maps bytes one-on-one to Unicode codepoints) or wrap your data into a representation that can be stored as text.
For Django 1.6 and up, use a BinaryField, for example. For earlier versions, using a binary-to-text conversion (such as Base64) would be preferable over decoding to Latin-1; the result of the latter would not give you meaningful textual data but Django may try to display it as such (in the admin interface for example).
It's occurring because you're attempting to save non-text data in a text field. Either use a non-text field instead, or encode the data as text via e.g. Base-64 encoding.
Using base64 encoding and decoding here fixed this:
import base64
def create_credential(self):
des = DES.new(CIPHER_N, DES.MODE_ECB)
text = str(uuid.uuid4()).replace('-','')[:16]
cipher_text = des.encrypt(text)
base64_encrypted_message = base64.b64encode(cipher_text)
return base64_encrypted_message
def decrypt_credential(self, text):
text = base64.b64decode(text)
des = DES.new(CIPHER_N, DES.MODE_ECB)
message = des.decrypt(text)
return message
I have a class chunk with text fields title and text. When I want to print them, I get (surprise, surprise!) UnicodeDecodeError. It gives me an error when I try to format an output string, but when I just concatenate text and title and return it, I get no error:
class Chunk:
# init, fields, ...
# this implementation will give me an error
def __str__( self ):
return u'{0} {1}'.format ( enc(self.text), enc(self.title) )
# but this is OK - all is printed without error
def __str__( self ):
return enc(self.text) + enc(self.title)
def enc(x):
return x.encode('utf-8','ignore') # tried many combinations of arguments...
c = Chunk()
c.text, c.title = ... # feed from external file
print c
Bum! Error!
return u'{0} {1}'.format ( enc(self.text), enc(self.title) )
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 2844: ordinal not in range(128)
I think I used all the possible combinations of encode/decode/utf-8/ascii/replace/ignore/...
(the python unicode issue is really irritating!)
You should override __unicode__, not __str__, when you return a unicode.
There is no need to call .encode(), since the input is already a unicode. Just write
def __unicode__(self):
return u"{0} {1}".format(self.text, self.title)
The simplest way to avoid 2.x python's unicode problem is to set overall encoding to utf-8, or such a problems will be constantly arise in a sudden places:
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
I have a text-box which allows users to enter a word.
The user enters: über
In the backend, I get the word like this:
def form_process(request):
word = request.GET.get('the_word')
word = word.encode('utf-8')
#word = word.decode('utf-8')
print word
For some reason, I cannot decode or encode this!!
It gives me the error:
UnicodeEncodeError
('ascii', u'\ufffd', 0, 1, 'ordinal not in range(128)')
Edit: When I do "repr(word)", this is what I get:
u'\ufffd'
Did you remember to put:
accept-charset="utf-8"
in the form tag?
EDIT: Is the DEFAULT_CHARSET in settings.py set to 'utf-8' ?
Solved!
I had escape(word) ...in the javascript ...before I passed it to the server.
Is there any reason to use print word? If not, its should work without those lines.
def form_process(request):
word = request.GET.get('the_word')