I have a strange problem with lxml when using the deployed version of my Django application. I use lxml to parse another HTML page which I fetch from my server. This works perfectly well on my development server on my own computer, but for some reason it gives me UnicodeDecodeError on the server.
('utf8', "\x85why hello there!", 0, 1, 'unexpected code byte')
I have made sure that Apache (with mod_python) runs with LANG='en_US.UTF-8'.
I've tried googling for this problem and tried different approaches to decoding the string correctly, but I can't figure it out.
In your answer, you may assume that my string is called hello or something.
"\x85why hello there!" is not a utf-8 encoded string. You should try decoding the webpage before passing it to lxml. Check what encoding it uses by looking at the http headers when you fetch the page maybe you find the problem there.
Doesn't syntax such as u"\x85why hello there!" help?
You may find the following resources from the official Python documentation helpful:
Python introduction, Unicode Strings
Sequence Types — str, unicode, list, tuple, buffer, xrange
Since modifying site.py is not an ideal solution try this at the start of your program:
import sys
reload(sys)
sys.setdefaultencoding("utf-8")
Related
device_id={UNIQUE_ID}&key={CUSTOMER_KEY}
I have tried following as many examples as I could find online, but there really wasn't a lot that was helpful for me. I'm using Python and have imported the msgpack and did this:
msgpack.packb([device_id:00000000-0000-0000-0000-000000000000,customer_key:0123456789abcdef])
But I get an error it's invalid syntax, with the colon indicated as the offending character. I wanted to out the msgpack code into a CURL and send it. But so far, I'm doing something wrong in how I define my key/values here.
I have a strange problem with converting special characters from HTML. I have a Django project where text is stored HTML-encoded in a MySQL database. This is necessary, because I don't want to lose any formatting of the text.
In a preliminary step I must do operational things on the text like calculating positions, so I need to convert it first and clear it from all HTML-Tags. This is done by BeautifulSoup:
convertedText = str(BeautifulSoup(text.text, convertEntities=BeautifulSoup.HTML_ENTITIES))
convertedText = ''.join(BeautifulSoup(convertedText).findAll(text=True))
By working on my Django-default test-server everything works fine, but when I run it on my production server there are strange behaviors when converting special characters.
An example:
Test server
MySQL-Query gives me: <p>bassverstärker</p>
is correctly converted to: bassverstärker
Production server
MySQL-Query gives me: <p>bassverstärker</p>
This is is wrongly converted to: bassverst\ucc44rker
Somehow the ä is converted into \ucc44 and this results in a wrong character.
My configuration:
Test server:
Django build-in solution (python manage.py runserver)
BeautifulSoup 3.2.1
Python 2.6.5
Ubuntu 2.6.32-43-generic
Production server:
Cherokee 1.2.101
BeautifulSoup 3.2.1
python 2.7.3
Ubuntu 3.2.0-32-generic
Because I don't know at which level the error occurs, I would like to ask if anybody can help me with this. Many thanks in advance.
I found a way to fix this. I didn't know that BeautifulSoup has the builtin method getText(). When converting HTML through:
convertedText = BeautifulSoup(text.text, convertEntities=BeautifulSoup.HTML_ENTITIES).getText()
eveything works fine on both servers. Although this works, it would be interesting to know why both servers are behaving differently when working with the example in the question.
However, thanks to all.
So I've been reading quite a bit about Unicoding tonight because I was thinking of switching to Jinja2, which requires Unicode to be used everywhere within the app. I think I have a good idea of how to deal with it, but I wanted to hear if this is reasonable before I started to code my app:
Dealing with External Text-Inputs (via html forms)
a) Make sure all html pages are utf-8 encoded.
b) Once users press submit, make sure the data is converted into Unicode as soon as the python backend receives it...decode(self.request.get('stuff'),utf-8)
c) Stay in unicode, transfer the outputs to Jinja2 which will always it using the default encoding of utf-8.
Information from the appengine datastore
Because google stores everything as Unicode, all data coming in from the datastore is already unicode and I don't have to worry about anything (yay!)
Strings within the app
Make sure all "" start with a u (i.e. u"hello world"), this will force everything to be in unicode.
Well the above is my strategy to keep everything consistent. Is there anything else I need to account for?
thanks!
You should not need to .decode(self.request.get('stuff'),utf-8 if you using webapp or webapp2. The framework respects the input type of the data as specified.
Everything else looks right.
Also I believe that
from __future__ import unicode_strings
should be
from __future__ import unicode_literals
and is only available in 2.6 and 2.7 So in App Engine it would only be available if you are using 2.7
I'm trying to do something similar to placekitten.com, wherein a user can input two strings after the base URL and have those strings alter the output. I'm doing this in Python, and I cannot for the life of me figure out how to grab the URL. In PHP I can do it with query string and $_REQUEST. I can't find a similar method in Python that doesn't rely on CGI.
(I know I could do this with Django, but that's serious overkill for this project.)
This is just by looking at the docs but have you tried it?
cherrypy.request.path_info
The docs say:
The ‘relative path’ portion of the Request-URI. This is relative to the script_name (‘mount point’) of the application which is handling this request.
http://docs.cherrypy.org/stable/refman/_cprequest.html#cherrypy._cprequest.Request.path_info
Hello StackOverflow community,
Using Google App Engine, I wrote a keyToSha256() method within a model class (extending db.Model) :
class Car(db.Model):
def keyToSha256(self):
keyhash = hashlib.sha256(str(self.key())).digest()
return keyhash
When displaying the output (ultimately within a Django template), I get garbled text, for example :
�����_ɘ�!`�I�!�;�QeqN��Al�'2
I was expecting something more in line with this :
9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
Am I missing something important ? Despite reading several guides on ASCII, Unicode, utf-8 and the like, I think I'm still far from mastering the secrets of string encoding/decoding. After browsing StackOverflow and searching for insights via Google, I figured out I should ask the question here. Any idea ? Thanks !
Use .hexdigest() instead.