I am running a python web app on an Ubuntu server, while I development locally on OS X.
I use a lot of unicode strings for the Hebrew language, including manipulating filenames of images, so they will be saved on the filesystem with Hebrew characters.
My Ubuntu server is fully configured for UTF-8 - I have other images on the file system (outside of this app) with Hebrew names, in Hebrew named directories, etc.
However, my app returns errors when trying to save an image with a Hebrew filename on Ubuntu (but not on OS X).
The error being:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
After alot of investigating, I got to the last possible cause as far as I can see:
# Inside my virtualenv, Mac OS X
>>> import os.path
>>> os.path.supports_unicode_filenames
>>> True
# Inside my virtualenv, Ubuntu 12.04
>>> import os.path
>>> os.path.supports_unicode_filenames
>>> False
And just for the curious, here are my Ubuntu locale settings:
locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Update: adding the code, and an example string:
# a string, of the type I would get for instance.product.name, as used below.
u'\\u05e7\\u05e8\\u05d5\\u05d1-\\u05e8\\u05d7\\u05d5\\u05e7'
#utils.py
# I get an image object from django, and I run this function so django
# can use the generated filepath for the image.
def get_upload_path(instance, filename):
tmp = filename.split('.')
extension = '.' + tmp[-1]
if instance.__class__.__name__ == 'MyClass':
seo_filename = unislugify(instance.product.name)
# unislugify takes a string and strips spaces, etc.
value = IMAGES_PRODUCT_DIR + seo_filename + extension
else:
value = IMAGES_GENERAL_DIR + unislugify(filename)
return value
Example stacktrace:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 60-66: ordinal not in range(128)
Stacktrace (most recent call last):
File "django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "django/contrib/admin/options.py", line 366, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "django/utils/decorators.py", line 91, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "django/views/decorators/cache.py", line 89, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "django/contrib/admin/sites.py", line 196, in inner
return view(request, *args, **kwargs)
File "django/utils/decorators.py", line 25, in _wrapper
return bound_func(*args, **kwargs)
File "django/utils/decorators.py", line 91, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "django/utils/decorators.py", line 21, in bound_func
return func(self, *args2, **kwargs2)
File "django/db/transaction.py", line 209, in inner
return func(*args, **kwargs)
File "django/contrib/admin/options.py", line 1055, in change_view
self.save_related(request, form, formsets, True)
File "django/contrib/admin/options.py", line 733, in save_related
self.save_formset(request, form, formset, change=change)
File "django/contrib/admin/options.py", line 721, in save_formset
formset.save()
File "django/forms/models.py", line 497, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)
File "django/forms/models.py", line 628, in save_new_objects
self.new_objects.append(self.save_new(form, commit=commit))
File "django/forms/models.py", line 731, in save_new
obj.save()
File "django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "django/db/models/query.py", line 1593, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "django/db/models/sql/compiler.py", line 909, in execute_sql
for sql, params in self.as_sql():
File "django/db/models/sql/compiler.py", line 872, in as_sql
for obj in self.query.objs
File "django/db/models/fields/files.py", line 249, in pre_save
file.save(file.name, file, save=False)
File "django/db/models/fields/files.py", line 86, in save
self.name = self.storage.save(name, content)
File "django/core/files/storage.py", line 44, in save
name = self.get_available_name(name)
File "django/core/files/storage.py", line 70, in get_available_name
while self.exists(name):
File "django/core/files/storage.py", line 230, in exists
return os.path.exists(self.path(name))
File "python2.7/genericpath.py", line 18, in exists
os.stat(path)
os.path.supports_unicode_filenames is always False on posix systems except darwin, that's because they don't really care about the encoding of the filename, it's simply a byte sequence. The locale settings specify how to interpret this bytes, that's why you can end up with broken characters in a terminal whenn the locale setting isn't right.
How are you running your web app? If your running it through a web server (apache?) using cgi or wsgi, the locale may not be what you see in the shell, so this could be the reason why python tries to use the ascii codec to encode the pathname.
To make it work, you could manually encode the pathname as utf-8 when opening the file.
Edit:
So the fails is a call to os.stat, which, wenn called with an unicode string, tries to convert it to a byte string according to the default encoding (sys.getdefaultencoding()), which within a uWSGI environment always seems to be ascii when using python2. To fix this you can make sure to encode any unicode string to utf-8 before it can be passed on to os.stat.
Thanks to the help of everyone. I still did not solve this issue with uWSGI.
But, this was the last straw in "configuring" uWSGI for me, I went back to gunicorn as the app server and everything works fine. I sure would like to use uWSGI as it is an ambitious project, but at the end of the day I am a developer and not a sys admin, and gunicorn is much easier to just get working in the common use cases.
Related
I am attempting to write a string directly into a Django FileField by way of ContentFile.
In doing so, I get a reproducible
TypeError: Unicode-objects must be encoded before hashing
error when attempting to save the contents of this file to the database, which traces through the s3boto3 lib.
The exact source of this error is difficult to suss out.
But let's state the question plainly, in Python 3, on Django 2.2.x, what is the correct way to take a csv file created with the csv lib from Python, and save that into a Django FileField backed by Amazon S3?
This question, and my approach, is inspired by this entry on SO Django - how to create a file and save it to a model's FileField? - however, given the age of the answer, some detail relevant to newer versions of Django appear to have been left out? Difficult to tell.
Example code producing the error in question, truncated for privacy and relevance
def campaign_to_csv_string(campaign_id):
csv_string = io.StringIO()
campaign = Campaign.objects.get(pk=campaign_id)
checklist = campaign.checklist
completed_jobs = JobRecord.objects.filter(appointment__campaign=campaign)
writer = csv.writer(csv_string)
# A bunch of writing to the writer here
# string looks good at this point
return csv_string.getvalue()
calling function
csv_string = campaign_to_csv_string(campaign_report.campaign.pk)
campaign_report.last_run = datetime.datetime.now()
campaign_report.report_file.save(str(campaign_report_pk) + '.report', ContentFile(csv_string))
campaign_report.processing = False
campaign_report.save()
My guess here is that s3boto3 is taking issue with ContentFile but the debugging information sent back to me gives me no clear path forward.
edit
Stack trace by request
TypeError: Unicode-objects must be encoded before hashing
File "celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "main/tasks.py", line 94, in produce_basic_campaign_report
campaign_report.report_file.save(str(campaign_report_pk) + '.report', csv_file)
File "django/db/models/fields/files.py", line 87, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "storages/backends/s3boto3.py", line 491, in _save
self._save_content(obj, content, parameters=parameters)
File "storages/backends/s3boto3.py", line 506, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "boto3/s3/inject.py", line 621, in object_upload_fileobj
ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
File "boto3/s3/inject.py", line 539, in upload_fileobj
return future.result()
File "s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "s3transfer/futures.py", line 265, in result
raise self._exception
File "s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "botocore/client.py", line 642, in _make_api_call
request_signer=self._request_signer, context=request_context)
File "botocore/hooks.py", line 360, in emit_until_response
return self._emitter.emit_until_response(aliased_event_name, **kwargs)
File "botocore/hooks.py", line 243, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
File "botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "botocore/handlers.py", line 212, in conditionally_calculate_md5
calculate_md5(params, **kwargs)
File "botocore/handlers.py", line 190, in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
File "botocore/handlers.py", line 204, in _calculate_md5_from_file
md5.update(chunk)
The csv string needs to be encoded as bytes when instantiating the ContentFile
The error can be reproduced this way:
from django.core.files.base import ContentFile
from botocore.handlers import _calculate_md5_from_file
_calculate_md5_from_file(ContentFile('throws error'))
TypeError: Unicode-objects must be encoded before hashing.
content isn't internal converted to bytes unless it is a gzip mimetype or explicitly compressed. https://github.com/jschneier/django-storages/blob/1.7.2/storages/backends/s3boto.py#L417
_calculate_md5_from_file is expecting a file containing bytes and this is the same for the underlying boto3 s3 client put_object method.
I suggest encoding csv_string as bytes.
campaign_report.report_file.save(
str(campaign_report_pk) + '.report',
ContentFile(
csv_string.encode()
)
)
On production server I receive the attached error, not emerging on the development server. Both environments are identical (same Django 1.6 and Python 2.7 versions, using virtualenvs same RDBMS version - a postgresql 9.1 server, running locally with a reasonably similar configuration than on production).
The offending code, I suppose, is in views.py:
from djgeojson.views import GeoJSONLayerView
class FilteredMapLayer(GeoJSONLayerView):
def get_queryset(self):
qs = super(FilteredMapLayer, self).get_queryset()
bbox_data = self.request.GET.get('bbox', None)
if bbox_data is not None:
bbox_data = ( x for x in map(float, bbox_data.split(',')))
bbox = Polygon.from_bbox(bbox_data)
features = qs.filter(geom__bboverlaps=bbox)
else:
features = qs
return features
since this is the urlpattern definition that, when requested, triggers the error
url(r'^data.geojson$', FilteredMapLayer.as_view(model=MyModelName, precision = 2, simplify = 0.5), name='myurl'),
here is the complete traceback, that does not provide me any info on where I did wrong....
Traceback (most recent call last):
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 114, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/utils/decorators.py", line 99, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/utils/decorators.py", line 29, in _wrapper
return bound_func(*args, **kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/utils/decorators.py", line 99, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in bound_func
return func(self, *args2, **kwargs2)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/djgeojson/views.py", line 51, in dispatch
return super(GeoJSONLayerView, self).dispatch(*args, **kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
return handler(request, *args, **kwargs)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/views/generic/list.py", line 153, in get
return self.render_to_response(context)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/djgeojson/views.py", line 41, in render_to_response
**options)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/djgeojson/serializers.py", line 335, in serialize
self.serialize_queryset(queryset)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/djgeojson/serializers.py", line 277, in serialize_queryset
self.handle_field(obj, self.geometry_field)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/djgeojson/serializers.py", line 177, in handle_field
geometry = self._handle_geom(GEOSGeometry(value))
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/djgeojson/serializers.py", line 159, in _handle_geom
geometry.transform(self.srid)
File "/home/m2user/webapps/enea/local/lib/python2.7/site-packages/django/contrib/gis/geos/geometry.py", line 510, in transform
raise GEOSException("Calling transform() with no SRID set is not supported")
.... since if I check the features stored in the model, they all have a SRID defined (4326) and I'm not trying to call any transformation - I suppose - since I'm rendering the view using the same SRID. And, I repeat myself here, the same request on dev server works seamlessly.
What I've tried so far:
recreated the DB from scratch, reloaded all the data. Using Django admin I can see/edit the features stored
checked all software dependences and updated the code from trunk of https://github.com/makinacorpus/django-geojson
checked model definitions: all have geom = models.MultiPolygonField(srid=4326) fields
searched Google and SO for similar issues without any luck
I have no more ideas. Any suggestion is welcome!
I was able to solve my issue by uninstalling and then re-installing all the libraries (as ubuntu packages), the python interpreter, all the code, by re-generating the database (including reinstalling postgis) and then by completely restart nginx (and thus gunicorn). No libraries were updated, however, so I still do not understand what really happened.
i am a newer in python.Today when I write some search function I met an error.well, I use sqlalchemy orm to do that, in my function,I input a chinese word as the key word.The html page give me an UnicodeEncodeError at /user/search:'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256).
and my code is like this:
def user_search(request):
name = request.GET.get('name').strip()
user_list = list()
if name:
user_list = User.get_by_name(name)
class User(object):
#classmethod
def get_by_name(cls, name):
return DBSession.query(cls).filter(cls.name==name)
and the Traceback is here:
Traceback:
File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py" in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File "/home/jiankong/git/admin-server/lib/decorators.py" in wrapper
75. return func(request, *args, **kwargs)
File "/home/jiankong/git/admin-server/lib/decorators.py" in wrapper
39. output = function(request, *args, **kwargs)
File "/home/jiankong/git/admin-server/apps/user/user_views.py" in user_search
47. users = jump_page(paginator, page)
File "/home/jiankong/git/admin-server/apps/user/utils.py" in jump_page
92. return paginator.page(1)
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in page
37. number = self.validate_number(number)
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in validate_number
28. if number > self.num_pages:
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in _get_num_pages
60. if self.count == 0 and not self.allow_empty_first_page:
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in _get_count
48. self._count = self.object_list.count()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in count
2414. return self.from_self(col).scalar()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in scalar
2240. ret = self.one()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in one
2209. ret = list(self)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in __iter__
2252. return self._execute_and_instances(context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in _execute_and_instances
2267. result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in execute
664. params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in _execute_clauseelement
764. compiled_sql, distilled_params
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in _execute_context
871. context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/default.py" in do_execute
324. cursor.execute(statement, parameters)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/cursors.py" in execute
183. query = query % db.literal(args)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/connections.py" in literal
264. return self.escape(o, self.encoders)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/connections.py" in unicode_literal
202. return db.literal(u.encode(unicode_literal.charset))
Exception Type: UnicodeEncodeError at /user/search
Exception Value: 'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256)`
when I met the error, I did a test in python shell, it worked well,the code is here:
from apps.user.models import User
user = User.get_by_name('某人').first()
print user
print user.name
某人
so what can I do to let it worked in my html page?much appreciate!!
I'm assuming that you're using MySQL with the MySQLdb driver here.
The default encoding used by the MySQLdb driver is latin-1, which does not support your character set. You'll need to use UTF-8 (or others, but UTF-8 is the most common) to be able to communicate with your database through MySQLdb (see http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#unicode).
To do such a thing, create your engine with the following line:
create_engine('mysql+mysqldb://USER:#SERVER:PORT/DB?charset=utf8', encoding='utf-8')
You can also construct your engine url using the sqlalchemy.engine.url.URL class, and send it to the create engine function. I find it useful when you have your settings in a config file.
import sqlalchemy.engine.url as url
engine_url = url.URL(
drivername='mysql+' + cfg['MYSQL_PYTHON_DRIVER'],
host=cfg['MYSQL_HOST'],
port=cfg['MYSQL_PORT'],
username=cfg['MYSQL_USER'],
password=cfg['MYSQL_PWD'],
database=cfg['MYSQL_DB'],
query={'charset': 'utf8'}
)
db = create_engine(engine_url, encoding='utf-8')
Hope that helps.
based on your stacktrace, you're using MySQL Python with unicode encoding turned on, since it's doing an encode. So you likely need to specify a comaptible encoding (note this is all settings used by the MySQLdb DBAPI, SQLalhcemy just passes them through):
create_engine('mysql+mysqldb:///mydb?charset=utf8&use_unicode=1')
http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#unicode
I have a project using gettext to translate English to Japanese but.
This is my code .py file
import gettext
t=gettext.translation('noname','advbus/locale', languages=['ja'])
_=t.gettext
print _("Hello")
and in file .po
msgid "Hello"
msgstr "今日は"
I compile .po file to .mop file but it doesn't display right 今日は.(utf8)
When i run .py file it run ok. But not right, my mean is it can't display unicode (今日�)
I try to set utf8 for file .mo file. But error
File "C:\env\Scripts\pserve-script.py", line 8, in <module>
load_entry_point('pyramid==1.4a1', 'console_scripts', 'pserve')()
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\scripts\pserve.py", line 47, in main
return command.run()
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\scripts\pserve.py", line 290, in run
relative_to=base, global_conf=vars)
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\scripts\pserve.py", line 318, in loadapp
return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 272, in loadobj
return context.create()
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "C:\env\lib\site-packages\paste\deploy\util.py", line 56, in fix_call
val = callable(*args, **kw)
File "C:\env\advbus\advbus\__init__.py", line 195, in main
config.scan()
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\config\__init__.py", line 946, in scan
ignore=ignore)
File "C:\env\lib\site-packages\venusian-1.0a7-py2.7.egg\venusian\__init__.py", line 197, in scan
__import__(modname)
File "C:\env\advbus\advbus\hello.py", line 3, in <module>
t=gettext.translation('noname','advbus/locale', languages=['ja'])
File "C:\env\lib\gettext.py", line 478, in translation
t = _translations.setdefault(key, class_(fp))
File "C:\env\lib\gettext.py", line 180, in __init__
self._parse(fp)
File "C:\env\lib\gettext.py", line 281, in _parse
raise IOError(0, 'Bad magic number', filename)
IOError: [Errno 0] Bad magic number: 'advbus/locale\\ja\\LC_MESSAGES\\noname.mo'
This is my subscribers.py file i coppy on the internet
from pyramid.i18n import get_localizer, TranslationStringFactory
def add_renderer_globals(event):
request = event['request']
event['_'] = request.translate
event['localizer'] = request.localizer
tsf = TranslationStringFactory('YOUR_GETTEXT_DOMAIN')
def add_localizer(event):
request = event.request
localizer = get_localizer(request)
def auto_translate(*args, **kwargs):
return localizer.translate(tsf(*args, **kwargs))
request.localizer = localizer
request.translate = auto_translate
The error message Bad magic number from gettext.py points to an improperly-formed .mo file. Read your own copy of Python's gettext.py, or look at this Apple-hosted copy of gettext.py.
I'm guessing the compiler from .po to .mo isn't working right. Show us your .po compile process, and maybe we can help further.
"Project-Id-Version: adv 0.0\n"
"Report-Msgid-Bugs-To: EMAIL#ADDRESS\n"
"POT-Creation-Date: 2012-12-01 11:41+0700\n"
"PO-Revision-Date: 2012-12-01 09:58+0700\n"
"Last-Translator: FULL NAME <EMAIL#ADDRESS>\n"
"Language-Team: ja <LL#li.org>\n"
"Plural-Forms: nplurals=1; plural=0\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 0.9.6\n"
#: advbus/hello.py:5
msgid "Hello"
msgstr "今日は"
That is my .po file
And my .mo file
今日ã¯
The string is not right
And i try run .py file, it works but the string same 今日ã¯
So i set utf-8 for .mo file, the string display on .mo file ok.
But when i run .py file, had error above
I have some code to parse an email and find the attachments then store them into the Datastore as db.BlobProperties (might change that to Blobstore later). The problem is that when I send a UTF8 encoded text file it generates an error.
The code basically saves the file and returns a key that is converted to a string then stored in the parent email entity. As you can see I decode the file then store it as a blob. I have sent lots of attachments and this code works on everything but Text encoded with Unicode. Is there a better way to do this? What can I do to handle Unicode text attachments?
code snipit
my_file = []
my_list = []
if hasattr(mail_message, 'attachments'):
file_name = ""
file_blob = ""
for filename, filecontents in mail_message.attachments:
file_name = filename
file_blob = filecontents.decode()
my_file.append(file_name)
my_list.append(str(store_file(self, file_name, file_blob)))
store_file
def store_file(self, file_name, file_blob):
new_file = myblob(file_name = file_name,
file_blob = file_blob)
return new_file.put()
I have tried using file_blob = str(file_blob) in the above to no avail. That just breaks the code and the file never gets stored.
log 1 of Unicode Text File
Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not unicode)
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/mail_handlers.py", line 65, in post
self.receive(mail.InboundEmailMessage(self.request.body))
File "/base/data/home/apps/s~ae-baseapp/1.359073377819595139/controllers/InboundHandler.py", line 51, in receive
file_list.append(str(store_file(self, file_name, file_blob)))
File "/base/data/home/apps/s~ae-baseapp/1.359073377819595139/models/MyModel.py", line 63, in store_file
file_blob = file_blob)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 974, in __init__
prop.__set__(self, value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
value = self.validate(value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2780, in validate
(self.name, self.data_type.__name__, err))
BadValueError: Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not unicode)
Log 2 of removing the filecontents.decode() and replacing it with just filecontents.
Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not EncodedPayload)
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/mail_handlers.py", line 65, in post
self.receive(mail.InboundEmailMessage(self.request.body))
File "/base/data/home/apps/s~ae-baseapp/1.359097282640216691/controllers/InboundHandler.py", line 57, in receive
file_list.append(str(store_file(self, file_name, file_blob)))
File "/base/data/home/apps/s~ae-baseapp/1.359097282640216691/models/MyModel.py", line 64, in store_file
file_blob = file_blob)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 974, in __init__
prop.__set__(self, value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
value = self.validate(value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2780, in validate
(self.name, self.data_type.__name__, err))
BadValueError: Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not EncodedPayload)
Attachment payloads are instances of the EncodedPayload class. Attachments have an encoding and an optional character set. The former refers to transfer encodings such as base64; the latter to character encodings such as UTF-8 (character set's a bit of an outdated and misleading term here). The EncodedPayload.decode() method decodes both transfer encoding and text encoding, which as you've noticed is not very helpful if you just want to get the original bytes the user attached to their message.
There's a number of approaches you can take here, but what I'd recommend is duplicating EncodedPayload's logic for decoding transfer encoding, which looks something like this:
if filecontents.encoding and filecontents.encoding.lower() != '7bit':
try:
payload = filecontents.payload.decode(filecontents.encoding)
except LookupError:
raise UnknownEncodingError('Unknown decoding %s.' % filecontents.encoding)
except (Exception, Error), e:
raise PayloadEncodingError('Could not decode payload: %s' % e)
else:
payload = filecontents.payload
Note that if the attachment was text, you need to include the character encoding when you store it, or there'll be no way to interpret it when you send it back to the user - the original text could have been encoded using any character encoding.
Likewise, you should also save the mimetype of the attachment if you can, but this doesn't appear to be exposed anywhere in the API. You might want to consider avoiding using the IncomingMessage class at all, and instead decoding the body of the POST request using Python's mime message module.
Check if this code helps:
===========================
my_file = []
my_list = []
if hasattr(mail_message, 'attachments'):
file_name = ""
for filename, filecontents in mail_message.attachments:
file_name = filename
file_blob = filecontents.payload
file_blob = file_blob.decode(filecontents.encoding)
my_file.append(file_name)
my_list.append(str(store_file(self, file_name, file_blob)))