What I'm doing is:
via javascript, reading the DOM of webpage
converting to json string
sending to python as ajax
in Python, json decoding the string into object
What I want is for any text that is part of the json to be in unicode to avoid any character issues. I used to use beautifulsoup for this:
from bs4 import *
from bs4.dammit import UnicodeDammit
text_unicode = UnicodeDammit(text, [None, None], "html", True).unicode_markup
But that doesn't work with the json string. Running the string through UnicodeDammit causes an error when I try to json decode it.
The thing is, I'm not even sure that collecting the DOM doesn't handle this issue automatically.
For starters, I would therefore like a series of test webpages to test this. Where one is encoded with utf-8, another with something else, etc. And that uses characters that will look wrong if, for example, you think it's utf-8 but it's not. Note that I don't even bother considering the webpage's stated encoding. This is too often wrong.
You are trying to solve a problem that does not exist.
The browser is responsible for detecting and handling the web page encoding. It'll determine the correct encoding based on the server headers, meta tags in the HTML page and plain guessing if needed. The DOM gives you Unicode data.
JSON handles Unicode data; sending JSON data to your Python process sends appropriately encoded byte data that any decent JSON library will turn back into Unicode values for you. The Python json module is such a library.
Just load the data from your JavaScript script with the json.load() or json.loads() functions as is. Your browser will already have used the correct encoding (most likely UTF-8), and the Python json module will decode any of the standard encodings used without additional configuration or handling.
Related
I may be poor at googling, but so far I have come up dry. Is there no such thing as a universal decoder for HTTP responses, where you give it the body and the headers, and it returns the decoded data?
For example:
response = requests.get("...")
body = clever_package.decode(response.body, response.headers)
This is using the requests package to get the data, though this isn't strictly necessary. Is there no universal decoder which takes the contentType and isBase64Encoded headers and works its magic?
Perhaps I'm not seeing an obvious flaw in such a package, which explains why I can't find it anywhere.
Cheers!
What do you mean with decoding? Just bytes to string data? In that case, python-chardet would be what you are looking for for cases where the header doesn't specify the encoding (if the headers specify the decoding, just decode it from the encoding specified in the header).
If you want to parse XML, JSON, ... in different ways, you'd probably use the respective libraries (built-in json module, yaml module, etc..) after having decoded the data into a unicode string.
I'm using python to retrieve an HTML source, but what comes out looks like this. What is this, and why am I not getting the actual page source?
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00C
This is an image. Specifically a jpeg. Since it's a byte stream python prints it with b'.............'
A jpeg starts with \xff\xd8\xff\
Try using BeautifulSoup
Here's an example
How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup?
Basically, what you're seeing is encoded characters that need to be decoded.
I'd like to scrape a website using Python that is full of horrible problems, one being the wrong encoding at the top:
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
This is wrong because the page is full of occurrences like the following:
Nell’ambito
instead of
Nell'ambito (please notice ’ replaces ')
If I understand correctly, this is happening because utf-8 bytes (probably the database encoding) are interpreted as iso-8859-1 bytes (forced by the charset in the meta tag).
I found some initial explanation at this link http://www.i18nqa.com/debug/utf8-debug.html
I am using BeautifulSoup to navigate the page, Google App Engine's urlfetch to make requests, however all I need is to understand what is the correct way to store in my database a string that fixes ’ by encoding the string to '.
I am using BeautifulSoup to navigate the page, Google App Engine's urlfetch to make requests
Are you feeding the encoding from the Content-Type HTTP header into BeautifulSoup?
If an HTML page has both a Content-Type header and a meta tag, the header should ‘win’, so if you're only taking the meta tag you may get the wrong encoding.
Otherwise, you could either feed the fixed encoding 'utf-8' into Beautiful, or fix up each string indvidually.
Annoying note: it's not actually ISO-8859-1. When web pages say ISO-8859-1, browsers actually take it to mean Windows code page 1252, which is similar to 8859-1 but not the same. The € would seem to indicate cp1252 because it's not present in 8859-1.
u'Nell’ambito'.encode('cp1252').decode('utf-8')
If the content is encoded inconsistently with some UTF-8 and some cp1252 on the same page (typically due to poor database content handling), this would be the only way to recover it, catching UnicodeError and returning the original string when it wouldn't transcode.
I have a script that needs to determine the charset before being read by lxml.HTML() for parsing. I will assume ISO-8859-1(that's the normal assumed charset for this right?) if it can't be found and search the html for the meta tag with the charset attribute. However I'm not sure the best way to do that. I could try to create an etree with lxml, but I don't want to read the whole file since I may run into encoding problems. However, if I don't read the whole file I can't build an etree since some tags will not have been closed.
Should I just find the meta tag with some fancy string subscripting and break out of the loop once it's found or a certain number of lines have been read? Maybe use a low level HTML parser, eg html.parser? Using python3 btw, thanks.
You should first try to extract encoding from HTTP headers. If it is not present there, you should parse it with the lxml. This might be tricky since lxml throws parse errors if charset does not match. A work-around would be decoding and encoding the data ignoring the unknown characters.
html_data=html_data.decode("UTF-8","ignore")
html_data=html_data.encode("UTF-8","ignore")
After this, you can parse by invoking the lxml.HTML() command with utf-8 encoding.
This way, you'll be able to find the correct encoding defined in the HTML headers.
After finding the encoding, you'll have to re-parse the HTML document with proper encoding.
Unfortunately, sometimes you might not find character encoding even in the HTML headers. I'd suggest you using the chardet module to find the proper encoding only after these steps fail.
Determining the character encoding of an HTML file correctly is actually quite a complex matter, but the HTML5 spec defines exactly how a processor should do it. You can find the algorithm here: http://dev.w3.org/html5/spec/parsing.html#determining-the-character-encoding
I'm reading some documentation on a service I'm trying to use, and it reads something like this:
All requests must be sent using HTTP Post.
The XML engine only accepts plain ASCII (text) UTF-8 requests/streams. Encoded streams are not acceptable.
All requests/responses are XML.
But I really just don't understand what it's asking for. From what I've been reading on HTTP POST in Python, you still need to encode key=value pairs to make a request, where it sounds like they just want the plain XML itself (as a multipart, maybe? I am very confused). Are they giving me enough information and I'm just fundamentally misunderstanding their documentation, or should I ask for more details?
using urllib2.Request
import urllib2
req = urllib2.Request("http://foo.com/post_here", "<xml data to post>")
response = urllib2.urlopen(req)
the_page = response.read()
"plain ASCII UTF-8" is a contradiction in terms, IMHO -- ASCII is a subset of UTF-8, though. Try sending UTF-8 including some "special" (non-ASCII) character and see what happens (or, if you can, do ask them to reword said contradition-in-terms!-).