I am porting my site to django 1.9 and don't know how to resolve this issue correctly.
In my form I have usual EmailField from django forms. If validation fails, it shoud be message about it (I pass 'form_errors': form.errors} to context for manipulating).
But in that case django returns
DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in
position 0: ordinal not in range(128). You passed in
()
in django.core.validators there is a validator for it
#deconstructible
class EmailValidator(object):
message = _('Enter a valid email address.')
...
If I change message to message = 'error' all works fine.
So, question: how can I fix this issue without editing django files?
the problem is in calling form.errors, this error raise even if I want only to print it. (print form.errors). Another fields errors (IntegerField, URLField, for example) works fine, this problem is only for EmailField.
in view process looks like that now:
from django.http import JsonResponse
...
if form.is_valid():
...
else:
return JsonResponse({'form_errors': form.errors})
last traceback is:
File "/path/views.py", line 331, in custom_form_post
response = JsonResponse({'form_errors': form.errors})
File "/path/.env/local/lib/python2.7/site-packages/django/http/response.py", line 505, in __init__
data = json.dumps(data, cls=encoder, **json_dumps_params)
File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "/path/.env/lib/python2.7/_abcoll.py", line 581, in __iter__
v = self[i]
File "/path/.env/local/lib/python2.7/site-packages/django/forms/utils.py", line 146, in __getitem__
return list(error)[0]
File "/path/.env/local/lib/python2.7/site-packages/django/core/exceptions.py", line 165, in __iter__
yield force_text(message)
File "/path/.env/local/lib/python2.7/site-packages/django/utils/encoding.py", line 88, in force_text
raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128). You passed in <django.utils.functional.__proxy__ object at 0x40a6a90c> (<class 'django.utils.functional.__proxy__'>)
Well. The problem is in translations. In this case - Russian localization.
No idea why translation from "native" django localization files failed.
But for all who have similar problem:
Create (if still not) locale file (https://docs.djangoproject.com/en/1.9/topics/i18n/translation/#localization-how-to-create-language-files)
Add theese rows to django.po:
msgid "Enter a valid email address."
msgstr "Введите правильный адрес электронной почты." (or another translation you need)
Compile (django-admin compilemessages)
Related
i have run into an encoding related issue lately in python and elasticsearch which i find is weird. What happens is i have a script which downloads some data from a an api it works fine on localhost. but when i try to run the same script on my server it throws an exception
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 143: ordinal not in range(128)
i understand this is beecause of some non-ascii characters present in the the data downloaded. but why then same script runs on my local without throwing any error?
i cannot post complete code because its a part of bigger application and it won't make sense on its own other than these:
def save_data(results,index,dimension):
def actionGenerator(results,index_name,doc_type_name):
for i in results:
yield {
"_index": index_name,
"_type": doc_type_name,
"_source": i
}
res = bulk(ES,actionGenerator(results,index,dimension),chunk_size=15000)
Is this error because of ES client? but in any case it works just fine on localhost
I guess exception is being thrown here looking at traceback:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/srv/fbweaver/src/main/tasks/task.py", line 189, in data_dump
parse_csv_file(FILE_NAME,REPORT_DOWNLOAD_PATH,breakdown.capitalize())
File "/srv/fbweaver/src/main/tasks/task.py", line 128, in parse_csv_file
save_data(result,'fb',breakdown)
File "/srv/fbweaver/src/main/tasks/task.py", line 22, in save_data
res = bulk(ES,actionGenerator(results,index,dimension),chunk_size=15000)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 190, in bulk
for ok, item in streaming_bulk(client, actions, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 162, in streaming_bulk
for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 87, in _process_bulk_chunk
resp = client.bulk('\n'.join(bulk_actions) + '\n', **kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 143: ordinal not in range(128)
I am using Robot Framework SSHLibrary to open connection with a RHEL server. But connection was unsuccessful. Robot Framework throws the following error
FAIL : UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 660: invalid start byte
20151212 12:47:36.022 : DEBUG :
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\SSHLibrary\library.py", line 792, in login
return self._login(self.current.login, username, password, delay)
File "C:\Python27\lib\site-packages\SSHLibrary\library.py", line 832, in _login
login_output = login_method(username, *args)
File "C:\Python27\lib\site-packages\SSHLibrary\abstractclient.py", line 150, in login
return self._read_login_output(delay)
File "C:\Python27\lib\site-packages\SSHLibrary\abstractclient.py", line 165, in _read_login_output
return self.read(delay)
File "C:\Python27\lib\site-packages\SSHLibrary\abstractclient.py", line 299, in read
return self._decode(output)
File "C:\Python27\lib\site-packages\SSHLibrary\abstractclient.py", line 302, in _decode
return output.decode(self.config.encoding)
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
In Open Connection Robot Framework keyword gave encoding=latin-1 as Latin character representation was present in my login response from remote server.
Login successful.
Environment: Python 2.7.4 (partly on Windows, partly on Linux, see below), suds (SVN HEAD with minor modifications)
I need to call into a web service that takes a single argument, which is an XML string (yes, I know…), i.e. the request is declared in the WSDL with the following type:
<s:complexType>
<s:sequence>
<s:element minOccurs="0" maxOccurs="1" name="actionString" type="s:string"/>
</s:sequence>
</s:complexType>
I'm using cElementTree to construct this inner XML document, then I pass it as the only parameter to the client.service.ProcessAction(request) method that suds generates.
For a while, this worked okay:
root = ET.Element(u'ActionCommand')
value = ET.SubElement(root, u'value')
value.text = saxutils.escape(complex_value)
request = u'<?xml version="1.0" encoding="utf-8"?>\n' + ET.tostring(root, encoding='utf-8')
client.service.ProcessAction(request)
The saxutils.escape, I had added at some point to fix the first encoding problems, pretty much without being able to understand why exactly I need it and what difference it makes.
Now (possibly due to the first occurence of the pound sign), I suddenly got the following exception:
Traceback (most recent call last):
File "/app/module.py", line 135, in _process_web_service_call
request = u'<?xml version="1.0" encoding="utf-8"?>\n' + ET.tostring(root, encoding='utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 137: ordinal not in range(128)
The position 137 here corresponds to the location of the special characters inside the inner XML request. Apparently, cElementTree.tostring() returns a 'str' type, not a 'unicode' even when an encoding is given. So Python tries to decode this string str into unicode (why with 'ascii'?), so that it can concatenate it with the unicode literal. This fails (of course, because the str is actually encoded in UTF-8, not ASCII).
So I figured, fine, I'll decode it to unicode myself then:
root = ET.Element(u'ActionCommand')
value = ET.SubElement(root, u'value')
value.text = saxutils.escape(complex_value)
request_encoded_str = ET.tostring(root, encoding='utf-8')
request_unicode = request_encoded_str.decode('utf-8')
request = u'<?xml version="1.0" encoding="utf-8"?>\n' + request_unicode
client.service.ProcessClientAction(request)
Except that now, it blows up inside suds, which tries to decode the outer XML request for some reason:
Traceback (most recent call last):
File "/app/module.py", line 141, in _process_web_service_call
raw_response = client.service.ProcessAction(request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/client.py", line 643, in send
reply = transport.send(request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/transport/https.py", line 64, in send
return HttpTransport.send(self, request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/transport/http.py", line 118, in send
return self.invoke(request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/transport/http.py", line 153, in invoke
u2response = urlopener.open(u2request, timeout=tm)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 1181, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 827, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 565: ordinal not in range(128)
The position 565 here again corresponds with the same character as above, except this time it's the location of my inner XML request embedded into the outer XML request (SOAP) created by suds.
I'm confused. Can anyone help me out of this mess? :)
To make matters worse, all of this only happens on the server under Linux. None of these raises an exception in my development environment on Windows. (Bonus points for an explanation as to why that is, just because I'm curious. I suspect it has to do with a different default encoding.) However, they all are not accepted by the server. What does work on Windows is if I drop the saxutils.escape and then hand a proper unicode object to suds. This however still results in the same UnicodeDecodeError on Linux.
Update: I started debugging this on Windows (where it works fine), and in the line 827 of httplib.py, it indeed tries to concatenate the unicode object msg (containing the HTTP headers) and the str object message_body, leading to the implicit unicode decoding with the incorrect encoding. I guess it just happens to not fail on Windows for some reason. I don't understand why suds tries to send a str object when I put a unicode object in at the top.
This turned out to be more than absurd. I'm still understanding only small parts of the whole problem and situation, but I managed to solve my problem.
So let's trace it back: my last attempt was the most sane one, I believe. So let's start there:
msg += message_body
That line in Python's httplib.py tries to concatenate a unicode and a str object, which leads to an implicit .decode('ascii') of the str, even though the str is UTF8-encoded. Why is that? Because msg is a unicode object.
msg = "\r\n".join(self._buffer)
self._buffer is a list of HTTP headers. Inspecting that, only one header in there was unicode, 'infecting' the resulting string: the action and endpoint.
And there's the problem: I'm using unicode_literals from __future__ (makes it more future-proof, right? right???) and I'm passing my own endpoint into suds.
By just doing an .encode('utf-8') on the URL, all my problems went away. Even the whole saxutils.escape was no longer needed (even though it weirdly also didn't hurt).
tl;dr: make sure you're not passing any unicode objects anywhere into httplib or suds, I guess.
root = ET.Element(u'ActionCommand')
value = ET.SubElement(root, u'value')
value.text = complex_value)
request = ET.tostring(root, encoding='utf-8').decode('utf-8')
client.service.ProcessAction(request)
i am a newer in python.Today when I write some search function I met an error.well, I use sqlalchemy orm to do that, in my function,I input a chinese word as the key word.The html page give me an UnicodeEncodeError at /user/search:'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256).
and my code is like this:
def user_search(request):
name = request.GET.get('name').strip()
user_list = list()
if name:
user_list = User.get_by_name(name)
class User(object):
#classmethod
def get_by_name(cls, name):
return DBSession.query(cls).filter(cls.name==name)
and the Traceback is here:
Traceback:
File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py" in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File "/home/jiankong/git/admin-server/lib/decorators.py" in wrapper
75. return func(request, *args, **kwargs)
File "/home/jiankong/git/admin-server/lib/decorators.py" in wrapper
39. output = function(request, *args, **kwargs)
File "/home/jiankong/git/admin-server/apps/user/user_views.py" in user_search
47. users = jump_page(paginator, page)
File "/home/jiankong/git/admin-server/apps/user/utils.py" in jump_page
92. return paginator.page(1)
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in page
37. number = self.validate_number(number)
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in validate_number
28. if number > self.num_pages:
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in _get_num_pages
60. if self.count == 0 and not self.allow_empty_first_page:
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in _get_count
48. self._count = self.object_list.count()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in count
2414. return self.from_self(col).scalar()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in scalar
2240. ret = self.one()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in one
2209. ret = list(self)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in __iter__
2252. return self._execute_and_instances(context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in _execute_and_instances
2267. result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in execute
664. params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in _execute_clauseelement
764. compiled_sql, distilled_params
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in _execute_context
871. context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/default.py" in do_execute
324. cursor.execute(statement, parameters)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/cursors.py" in execute
183. query = query % db.literal(args)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/connections.py" in literal
264. return self.escape(o, self.encoders)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/connections.py" in unicode_literal
202. return db.literal(u.encode(unicode_literal.charset))
Exception Type: UnicodeEncodeError at /user/search
Exception Value: 'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256)`
when I met the error, I did a test in python shell, it worked well,the code is here:
from apps.user.models import User
user = User.get_by_name('某人').first()
print user
print user.name
某人
so what can I do to let it worked in my html page?much appreciate!!
I'm assuming that you're using MySQL with the MySQLdb driver here.
The default encoding used by the MySQLdb driver is latin-1, which does not support your character set. You'll need to use UTF-8 (or others, but UTF-8 is the most common) to be able to communicate with your database through MySQLdb (see http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#unicode).
To do such a thing, create your engine with the following line:
create_engine('mysql+mysqldb://USER:#SERVER:PORT/DB?charset=utf8', encoding='utf-8')
You can also construct your engine url using the sqlalchemy.engine.url.URL class, and send it to the create engine function. I find it useful when you have your settings in a config file.
import sqlalchemy.engine.url as url
engine_url = url.URL(
drivername='mysql+' + cfg['MYSQL_PYTHON_DRIVER'],
host=cfg['MYSQL_HOST'],
port=cfg['MYSQL_PORT'],
username=cfg['MYSQL_USER'],
password=cfg['MYSQL_PWD'],
database=cfg['MYSQL_DB'],
query={'charset': 'utf8'}
)
db = create_engine(engine_url, encoding='utf-8')
Hope that helps.
based on your stacktrace, you're using MySQL Python with unicode encoding turned on, since it's doing an encode. So you likely need to specify a comaptible encoding (note this is all settings used by the MySQLdb DBAPI, SQLalhcemy just passes them through):
create_engine('mysql+mysqldb:///mydb?charset=utf8&use_unicode=1')
http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#unicode
I'm crawling pages and indexing them with appengine search api (Spanish and Catalan pages, with accented characters). I'm able to perform searches and make a page of results.
Problem arises when I try to use a query object with snipetted_fields, as it always generates a UnicodeEncodeError:
File "/home/otger/python/jobs-gae/src/apps/search/handlers/results.py", line 82, in find_documents
return index.search(query_obj)
File "/opt/google_appengine_1.7.6/google/appengine/api/search/search.py", line 2707, in search
apiproxy_stub_map.MakeSyncCall('search', 'Search', request, response)
File "/opt/google_appengine_1.7.6/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "/opt/google_appengine_1.7.6/google/appengine/api/apiproxy_stub_map.py", line 320, in MakeSyncCall
rpc.CheckSuccess()
File "/opt/google_appengine_1.7.6/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "/opt/google_appengine_1.7.6/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/opt/google_appengine_1.7.6/google/appengine/ext/remote_api/remote_api_stub.py", line 234, in _MakeRealSyncCall
raise pickle.loads(response_pb.exception())
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 52: ordinal not in range(128)
I've found a similar question on stackoverflow: GAE Full Text Search development console UnicodeEncodeError but it says that it was a bug fixed on 1.7.0. I get same error either using version 1.7.5 and 1.7.6.
When Indexing pages I add two fields: description and description_ascii. If I try to generate snippets for description_ascii it works perfectly.
Is this possible to generate snippets of not ascii contents on dev_appserver?
I think this is a bug, reported new defect issue https://code.google.com/p/googleappengine/issues/detail?id=9335.
Temporary solution for dev server - locate google.appengine.api.search module (search.py), and patch function _DecodeUTF8 by adding inline if like this:
def _DecodeUTF8(pb_value):
"""Decodes a UTF-8 encoded string into unicode."""
if pb_value is not None:
return pb_value.decode('utf-8') if not isinstance(pb_value, unicode) else pb_value
return None
Workaround - until the issue is solved implement snippet functionality yourself - assuming field which is base for snippet is called snippet_base:
query = search.Query(query_string=query_string,
options=
search.QueryOptions(
...
returned_fields= [... 'snippet_base' ...]
))
results = search.Index(name="<index-name>").search(query)
if results:
for res in results.results:
res.snippet = some_snippeting_function(res.field("snippet_base"))