I have a project using gettext to translate English to Japanese but.
This is my code .py file
import gettext
t=gettext.translation('noname','advbus/locale', languages=['ja'])
_=t.gettext
print _("Hello")
and in file .po
msgid "Hello"
msgstr "今日は"
I compile .po file to .mop file but it doesn't display right 今日は.(utf8)
When i run .py file it run ok. But not right, my mean is it can't display unicode (今日�)
I try to set utf8 for file .mo file. But error
File "C:\env\Scripts\pserve-script.py", line 8, in <module>
load_entry_point('pyramid==1.4a1', 'console_scripts', 'pserve')()
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\scripts\pserve.py", line 47, in main
return command.run()
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\scripts\pserve.py", line 290, in run
relative_to=base, global_conf=vars)
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\scripts\pserve.py", line 318, in loadapp
return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 272, in loadobj
return context.create()
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "C:\env\lib\site-packages\paste\deploy\loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "C:\env\lib\site-packages\paste\deploy\util.py", line 56, in fix_call
val = callable(*args, **kw)
File "C:\env\advbus\advbus\__init__.py", line 195, in main
config.scan()
File "C:\env\lib\site-packages\pyramid-1.4a1-py2.7.egg\pyramid\config\__init__.py", line 946, in scan
ignore=ignore)
File "C:\env\lib\site-packages\venusian-1.0a7-py2.7.egg\venusian\__init__.py", line 197, in scan
__import__(modname)
File "C:\env\advbus\advbus\hello.py", line 3, in <module>
t=gettext.translation('noname','advbus/locale', languages=['ja'])
File "C:\env\lib\gettext.py", line 478, in translation
t = _translations.setdefault(key, class_(fp))
File "C:\env\lib\gettext.py", line 180, in __init__
self._parse(fp)
File "C:\env\lib\gettext.py", line 281, in _parse
raise IOError(0, 'Bad magic number', filename)
IOError: [Errno 0] Bad magic number: 'advbus/locale\\ja\\LC_MESSAGES\\noname.mo'
This is my subscribers.py file i coppy on the internet
from pyramid.i18n import get_localizer, TranslationStringFactory
def add_renderer_globals(event):
request = event['request']
event['_'] = request.translate
event['localizer'] = request.localizer
tsf = TranslationStringFactory('YOUR_GETTEXT_DOMAIN')
def add_localizer(event):
request = event.request
localizer = get_localizer(request)
def auto_translate(*args, **kwargs):
return localizer.translate(tsf(*args, **kwargs))
request.localizer = localizer
request.translate = auto_translate
The error message Bad magic number from gettext.py points to an improperly-formed .mo file. Read your own copy of Python's gettext.py, or look at this Apple-hosted copy of gettext.py.
I'm guessing the compiler from .po to .mo isn't working right. Show us your .po compile process, and maybe we can help further.
"Project-Id-Version: adv 0.0\n"
"Report-Msgid-Bugs-To: EMAIL#ADDRESS\n"
"POT-Creation-Date: 2012-12-01 11:41+0700\n"
"PO-Revision-Date: 2012-12-01 09:58+0700\n"
"Last-Translator: FULL NAME <EMAIL#ADDRESS>\n"
"Language-Team: ja <LL#li.org>\n"
"Plural-Forms: nplurals=1; plural=0\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 0.9.6\n"
#: advbus/hello.py:5
msgid "Hello"
msgstr "今日は"
That is my .po file
And my .mo file
今日ã¯
The string is not right
And i try run .py file, it works but the string same 今日ã¯
So i set utf-8 for .mo file, the string display on .mo file ok.
But when i run .py file, had error above
Related
I am attempting to write a string directly into a Django FileField by way of ContentFile.
In doing so, I get a reproducible
TypeError: Unicode-objects must be encoded before hashing
error when attempting to save the contents of this file to the database, which traces through the s3boto3 lib.
The exact source of this error is difficult to suss out.
But let's state the question plainly, in Python 3, on Django 2.2.x, what is the correct way to take a csv file created with the csv lib from Python, and save that into a Django FileField backed by Amazon S3?
This question, and my approach, is inspired by this entry on SO Django - how to create a file and save it to a model's FileField? - however, given the age of the answer, some detail relevant to newer versions of Django appear to have been left out? Difficult to tell.
Example code producing the error in question, truncated for privacy and relevance
def campaign_to_csv_string(campaign_id):
csv_string = io.StringIO()
campaign = Campaign.objects.get(pk=campaign_id)
checklist = campaign.checklist
completed_jobs = JobRecord.objects.filter(appointment__campaign=campaign)
writer = csv.writer(csv_string)
# A bunch of writing to the writer here
# string looks good at this point
return csv_string.getvalue()
calling function
csv_string = campaign_to_csv_string(campaign_report.campaign.pk)
campaign_report.last_run = datetime.datetime.now()
campaign_report.report_file.save(str(campaign_report_pk) + '.report', ContentFile(csv_string))
campaign_report.processing = False
campaign_report.save()
My guess here is that s3boto3 is taking issue with ContentFile but the debugging information sent back to me gives me no clear path forward.
edit
Stack trace by request
TypeError: Unicode-objects must be encoded before hashing
File "celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "main/tasks.py", line 94, in produce_basic_campaign_report
campaign_report.report_file.save(str(campaign_report_pk) + '.report', csv_file)
File "django/db/models/fields/files.py", line 87, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "storages/backends/s3boto3.py", line 491, in _save
self._save_content(obj, content, parameters=parameters)
File "storages/backends/s3boto3.py", line 506, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "boto3/s3/inject.py", line 621, in object_upload_fileobj
ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
File "boto3/s3/inject.py", line 539, in upload_fileobj
return future.result()
File "s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "s3transfer/futures.py", line 265, in result
raise self._exception
File "s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "botocore/client.py", line 642, in _make_api_call
request_signer=self._request_signer, context=request_context)
File "botocore/hooks.py", line 360, in emit_until_response
return self._emitter.emit_until_response(aliased_event_name, **kwargs)
File "botocore/hooks.py", line 243, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
File "botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "botocore/handlers.py", line 212, in conditionally_calculate_md5
calculate_md5(params, **kwargs)
File "botocore/handlers.py", line 190, in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
File "botocore/handlers.py", line 204, in _calculate_md5_from_file
md5.update(chunk)
The csv string needs to be encoded as bytes when instantiating the ContentFile
The error can be reproduced this way:
from django.core.files.base import ContentFile
from botocore.handlers import _calculate_md5_from_file
_calculate_md5_from_file(ContentFile('throws error'))
TypeError: Unicode-objects must be encoded before hashing.
content isn't internal converted to bytes unless it is a gzip mimetype or explicitly compressed. https://github.com/jschneier/django-storages/blob/1.7.2/storages/backends/s3boto.py#L417
_calculate_md5_from_file is expecting a file containing bytes and this is the same for the underlying boto3 s3 client put_object method.
I suggest encoding csv_string as bytes.
campaign_report.report_file.save(
str(campaign_report_pk) + '.report',
ContentFile(
csv_string.encode()
)
)
I'm not sure if this is an issue with flask or an issue with matplotlib. I've been able to access images in html and javascript, but now I can't seem to do it in python.
I have a directory tree structure like the following
\---Neuroethics_Behavioral_Task # 'root' directory for my flask app. Contents are shown below
+---jspsych-6.0
| +---css
| ## cutting this directory off here, jspsych is just a framework I'm using, not relevant
+---static
| +---images
| +---plugins
| | \---template
| +---textdata
| \---trial_stimuli
+---templates
\---__pycache__
I've just been using a script called experiment.py that lives in the topmost directory of my flask app (so in the /Neuroethics_Behavioral_Task/ folder). It executes a function called "generate_stim_for_trials(trials)". Create_stimuli uses matplotlib to create a bunch of pngs, each representing stimuli, and it's supposed to save these images in the /static/images/ directory.
I'm omitting most of the code related to the stimulus creation for... "ownership/copyright" purposes. Also that code really shouldn't be relevant All that needs to be said is that I create a pandas dataframe and plot it in matplotlib.
def generate_stim_for_trials(trials):
import matplotlib
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import json
import os
lst_of_jsons_for_stimuli = []
stim_metadata_path = "/static/textdata/stimulus_data.json"
for each in range(trials):
stimuli_in_static = "images/stimulus_img" + str(each) + ".png"
stimuli_path = url_for('static', filename=stimuli_in_static)
# started ommitting code here...
df = # df variable gets created
df.plot(kind='barh', legend = False, stacked=True, color=['#e26a6a', '#3CB371', '#A9A9A9'])
plt.axis('off')
plt.margins(x=0)
plt.margins(y=0)
plt.grid(b=None)
plt.savefig(stimuli_path, bbox_inches='tight', pad_inches=-1) # error
The error I get is ultimately: FileNotFoundError: [Errno 2] No such file or directory: '/static/images/stimulus_img0.png'. I have the full error message but it's pretty long (and if moderators don't think it's useful, feel free to remove it/tell me to remove it).
Traceback (most recent call last):
File "D:\miniconda3\lib\site-packages\flask\app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "D:\miniconda3\lib\site-packages\flask\app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1866, in handle_exce
ption
reraise(exc_type, exc_value, tb)
File "D:\miniconda3\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "D:\miniconda3\lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1951, in full_dispat
ch_request
rv = self.handle_user_exception(e)
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1820, in handle_user
_exception
reraise(exc_type, exc_value, tb)
File "D:\miniconda3\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1949, in full_dispat
ch_request
rv = self.dispatch_request()
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1935, in dispatch_re
quest
return self.view_functions[rule.endpoint](**req.view_args)
File "D:\Projects\CogNeW\Neuroethics_Behavioral_Task\experiment.py", line 21,
in exp_js_psych_app
generate_stim_for_trials(total_trials)
File "D:\Projects\CogNeW\Neuroethics_Behavioral_Task\experiment.py", line 93,
in generate_stim_for_trials
plt.savefig(stimuli_path, bbox_inches='tight', pad_inches=-1)
File "D:\miniconda3\lib\site-packages\matplotlib\pyplot.py", line 722, in save
fig
res = fig.savefig(*args, **kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\figure.py", line 2180, in sav
efig
self.canvas.print_figure(fname, **kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\backends\backend_qt5agg.py",
line 88, in print_figure
super().print_figure(*args, **kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\backend_bases.py", line 2082,
in print_figure
**kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\backends\backend_agg.py", lin
e 530, in print_png
cbook.open_file_cm(filename_or_obj, "wb") as fh:
File "D:\miniconda3\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "D:\miniconda3\lib\site-packages\matplotlib\cbook\__init__.py", line 447,
in open_file_cm
fh, opened = to_filehandle(path_or_file, mode, True, encoding)
File "D:\miniconda3\lib\site-packages\matplotlib\cbook\__init__.py", line 432,
in to_filehandle
fh = open(fname, flag, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/static/images/stimulus
_img0.png'
Again, this python file experiment.py exists at the topmost level of the flask module, in the "neuroethics_behavioral_task" directory. So it should be able to understand the path, /static/images since the static folder is also at that same level. But it doesn't.
Furthermore, here I'm trying to create/save a new file called stimulus_img0.png. So indeed, this file shouldn't exist yet -- it's getting created.
I can't tell if this is a matplotlib or flask issue, and I"m not sure how to resolve it because I'm not sure where my python program is actually looking at this point.
Edit 1
Well the 'mystery' is solved.
You know how you're supposed to do flask run or python -m flask run from a directory above your flask app?
If I just make the line plt.savefig('stimuli_img0.png'), the image(s) get saved in the directory above my flask application, aka the directory from which I'm executing python -m flask run.
So now my question is... What actually controls where python scripts in Flask save to? Is this actually the default behavior? I haven't encountered this in the html or js files in this flask app.
I'm learning python (from a very low baseline) and recently re-installed Canopy (on a MacBook) It was working fine before.
Now whenever I try an launch the editor I get a Access Denied error.
Can anyone help? Please bear in mind my inexperience
Thanks
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/envisage/ui/tasks/tasks_application.py", line 205, in create_window
window.add_task(task)
File "/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/lib/python2.7/site-packages/pyface/tasks/task_window.py", line 187, in add_task
state.dock_panes.append(dock_pane_factory(task=task))
File "build/bdist.macosx-10.5-i386/egg/canopy/plugin/editor_task.py", line 143, in _create_python_pane
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/envisage/application.py", line 371, in get_service
protocol, query, minimize, maximize
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/envisage/service_registry.py", line 78, in get_service
services = self.get_services(protocol, query, minimize, maximize)
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/envisage/service_registry.py", line 115, in get_services
actual_protocol, name, obj, properties, service_id
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/envisage/service_registry.py", line 259, in _resolve_factory
obj = obj(**properties)
File "build/bdist.macosx-10.5-i386/egg/canopy/python_frontend/plugin.py", line 109, in _frontend_manager_service_factory
File "build/bdist.macosx-10.5-i386/egg/canopy/app/running_process_manager.py", line 82, in register_proc
File "build/bdist.macosx-10.5-i386/egg/canopy/app/util.py", line 53, in get_exe_or_cmdline
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/psutil/_common.py", line 80, in get
ret = self.func(instance)
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/psutil/init.py", line 331, in exe
return guess_it(fallback=err)
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/psutil/init.py", line 314, in guess_it
cmdline = self.cmdline
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/psutil/init.py", line 346, in cmdline
return self._platform_impl.get_process_cmdline()
File "/Users/simonthompson/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/psutil/_psosx.py", line 153, in wrapper
raise AccessDenied(self.pid, self._process_name)
AccessDenied: (pid=343)
DEBUG|2013-11-03 21:19:25|QtWarningMsg: QImage::scaled: Image is a null image
Since the supplied information is insufficient, the answer is the same. This is about user authentication. I don't know how you open the app but, your app tries to open a file or a process which is could not be opened by your user. If you open your app with root privileges there won't be any problem.
I am running a python web app on an Ubuntu server, while I development locally on OS X.
I use a lot of unicode strings for the Hebrew language, including manipulating filenames of images, so they will be saved on the filesystem with Hebrew characters.
My Ubuntu server is fully configured for UTF-8 - I have other images on the file system (outside of this app) with Hebrew names, in Hebrew named directories, etc.
However, my app returns errors when trying to save an image with a Hebrew filename on Ubuntu (but not on OS X).
The error being:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
After alot of investigating, I got to the last possible cause as far as I can see:
# Inside my virtualenv, Mac OS X
>>> import os.path
>>> os.path.supports_unicode_filenames
>>> True
# Inside my virtualenv, Ubuntu 12.04
>>> import os.path
>>> os.path.supports_unicode_filenames
>>> False
And just for the curious, here are my Ubuntu locale settings:
locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Update: adding the code, and an example string:
# a string, of the type I would get for instance.product.name, as used below.
u'\\u05e7\\u05e8\\u05d5\\u05d1-\\u05e8\\u05d7\\u05d5\\u05e7'
#utils.py
# I get an image object from django, and I run this function so django
# can use the generated filepath for the image.
def get_upload_path(instance, filename):
tmp = filename.split('.')
extension = '.' + tmp[-1]
if instance.__class__.__name__ == 'MyClass':
seo_filename = unislugify(instance.product.name)
# unislugify takes a string and strips spaces, etc.
value = IMAGES_PRODUCT_DIR + seo_filename + extension
else:
value = IMAGES_GENERAL_DIR + unislugify(filename)
return value
Example stacktrace:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 60-66: ordinal not in range(128)
Stacktrace (most recent call last):
File "django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "django/contrib/admin/options.py", line 366, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "django/utils/decorators.py", line 91, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "django/views/decorators/cache.py", line 89, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "django/contrib/admin/sites.py", line 196, in inner
return view(request, *args, **kwargs)
File "django/utils/decorators.py", line 25, in _wrapper
return bound_func(*args, **kwargs)
File "django/utils/decorators.py", line 91, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "django/utils/decorators.py", line 21, in bound_func
return func(self, *args2, **kwargs2)
File "django/db/transaction.py", line 209, in inner
return func(*args, **kwargs)
File "django/contrib/admin/options.py", line 1055, in change_view
self.save_related(request, form, formsets, True)
File "django/contrib/admin/options.py", line 733, in save_related
self.save_formset(request, form, formset, change=change)
File "django/contrib/admin/options.py", line 721, in save_formset
formset.save()
File "django/forms/models.py", line 497, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)
File "django/forms/models.py", line 628, in save_new_objects
self.new_objects.append(self.save_new(form, commit=commit))
File "django/forms/models.py", line 731, in save_new
obj.save()
File "django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "django/db/models/query.py", line 1593, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "django/db/models/sql/compiler.py", line 909, in execute_sql
for sql, params in self.as_sql():
File "django/db/models/sql/compiler.py", line 872, in as_sql
for obj in self.query.objs
File "django/db/models/fields/files.py", line 249, in pre_save
file.save(file.name, file, save=False)
File "django/db/models/fields/files.py", line 86, in save
self.name = self.storage.save(name, content)
File "django/core/files/storage.py", line 44, in save
name = self.get_available_name(name)
File "django/core/files/storage.py", line 70, in get_available_name
while self.exists(name):
File "django/core/files/storage.py", line 230, in exists
return os.path.exists(self.path(name))
File "python2.7/genericpath.py", line 18, in exists
os.stat(path)
os.path.supports_unicode_filenames is always False on posix systems except darwin, that's because they don't really care about the encoding of the filename, it's simply a byte sequence. The locale settings specify how to interpret this bytes, that's why you can end up with broken characters in a terminal whenn the locale setting isn't right.
How are you running your web app? If your running it through a web server (apache?) using cgi or wsgi, the locale may not be what you see in the shell, so this could be the reason why python tries to use the ascii codec to encode the pathname.
To make it work, you could manually encode the pathname as utf-8 when opening the file.
Edit:
So the fails is a call to os.stat, which, wenn called with an unicode string, tries to convert it to a byte string according to the default encoding (sys.getdefaultencoding()), which within a uWSGI environment always seems to be ascii when using python2. To fix this you can make sure to encode any unicode string to utf-8 before it can be passed on to os.stat.
Thanks to the help of everyone. I still did not solve this issue with uWSGI.
But, this was the last straw in "configuring" uWSGI for me, I went back to gunicorn as the app server and everything works fine. I sure would like to use uWSGI as it is an ambitious project, but at the end of the day I am a developer and not a sys admin, and gunicorn is much easier to just get working in the common use cases.
I'm trying to use django-pipeline-1.1.27 with s3boto to compress and filter static files, and then upload them to an s3 bucket. If I just use:
PIPELINE_STORAGE = 'pipeline.storage.PipelineFinderStorage'
Then it works and I get a static folder with the nice versioned file that I configured. As soon as I switch to
PIPELINE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
I get
Traceback (most recent call last):
File "manage.py", line 15, in <module>
execute_manager(settings)
File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
output = self.handle(*args, **options)
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/management/commands/synccompress.py", line 39, in handle
packager.pack_stylesheets(package, sync=sync, force=force)
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/packager.py", line 52, in pack_stylesheets
**kwargs)
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/packager.py", line 60, in pack
package['output'], package['paths'])
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/__init__.py", line 45, in need_update
version = self.version(paths)
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/__init__.py", line 20, in version
return getattr(self.versioner, 'version')(paths)
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 37, in version
buf = self.concatenate(paths)
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 27, in concatenate
return '\n'.join([self.read_file(path) for path in paths])
File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 31, in read_file
file = storage.open(path, 'rb')
File "/my/virtual/env/lib/python2.7/site-packages/django/core/files/storage.py", line 33, in open
file = self._open(name, mode)
File "/my/virtual/env/lib/python2.7/site-packages/storages/backends/s3boto.py", line 177, in _open
raise IOError('File does not exist: %s' % name)
IOError: File does not exist: css/style.css
which is one of my source files. So why does pipeline no longer want to do the filter/concatenate/compress steps when I switch to s3boto storage?
It may be that I'm doing something. Here is other config in case it helps:
INSTALLED_APPS = (
...
'pipeline',
'storages',
)
STATICFILES_FINDERS = (
'pipeline.finders.PipelineFinder',
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
STATIC_ROOT = "/some/path/outside/django_project/deploy_static"
STATICFILES_DIRS = () # All statics in this site are in apps
STATICFILES_STORAGE = 'pipeline.storage.PipelineStorage'
PIPELINE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
PIPELINE = True
PIPELINE_AUTO = True
PIPELINE_VERSION = True
PIPELINE_VERSION_PLACEHOLDER = 'VERSION'
PIPELINE_VERSIONING = 'pipeline.versioning.hash.SHA1Versioning'
PIPELINE_CSS = {
'standard': {
'source_filenames': (
'css/style.css',
...
),
'output_filename': 'css/all-VERSION.css',
'extra_context': {
'media': 'screen,projection',
},
}
}
My site is on Django 1.3.1.
The command I'm running is:
python manage.py synccompress --force
The AWS creds are also in settings, but that's moot because it's not even getting to that point.
UPDATE Added full stack and settings requested in comments
UPDATE At the request of the library author, I tried upgrading to the latest beta. Observations from that so far:
I don't know how to get versioned compressed files now
collectstatic leaves me with the compressed files and the originals
Still getting the same error from django-pipeline when boto storage is configured: it wants to send my source files to s3, but I can't even see where it's staging my assets. Nothing gets placed in STATIC_ROOT.
UPDATE I've created the simplest project that works for finder storage and then breaks with S3Boto. I've pushed it to github, and included a capture of the stacktrace.
https://github.com/estebistec/simple_pipeline
https://raw.github.com/estebistec/simple_pipeline/master/STACKTRACE
I would be ecstatic if I could be told I'm doing some really dumb and this should all just work.
django-pipeline 1.1.x is a bit dumb about how you should use staticfiles, it prefers to have everything in one place.
I suggest you to try django-pipeline 1.2 with latest django-staticfiles or django 1.4.
Use a custom like this :
STATICFILES_STORAGE = 'your.app.S3PipelineStorage'
The code looks like this :
from staticfiles.storage import CachedFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage
class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
pass
You can find how to fix your application, but there is still a bug with compiled files unless you use version 1.2c1 : https://gist.github.com/1999564
I just experienced this same error on a Django 1.6 project with django-pipeline==1.3.23, and the solution was simply removing the PIPELINE_STORAGE setting.
There is another problem with similar error message that affects earlier and current version (1.5.4) of django-pipeline.
The error message is IOError: File does not exist, and it happens in s3boto.py.open() and packager.pack_stylesheets(). You might hit the problem if you use any of the compiler (Compass, Sass, Less etc). I suspect it would also affect JS compiler, but I have not confirmed.
In a nutshell, the compiler generates output file in the local static storage and the next steps, compress is trying to find the output in s3 storage.
If it affects you, you might want to take a look at https://github.com/cyberdelia/django-pipeline/issues/473. There are two pull-requests (patches), one made by skirsdeda and another made by thomasyip (me). Both might solve your problem. If you would like the compiled (but, pre-compressed) file to be copied to s3 and available to the app, the you would take thomasyip (me)'s patch.
Here is full Traceback for the problem:
Traceback (most recent call last):
File "apps/manage.py", line 16, in <module>
execute_from_command_line(sys.argv)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 533, in handle
return self.handle_noargs(**options)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 171, in handle_noargs
collected = self.collect()
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 117, in collect
for original_path, processed_path, processed in processor:
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/storage.py", line 26, in post_process
packager.pack_stylesheets(package)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
variant=package.variant, **kwargs)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/packager.py", line 106, in pack
content = compress(paths, **kwargs)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 73, in compress_css
css = self.concatenate_and_rewrite(paths, output_filename, variant)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 137, in concatenate_and_rewrite
content = self.read_text(path)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 220, in read_text
content = self.read_bytes(path)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 214, in read_bytes
file = staticfiles_storage.open(path)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/files/storage.py", line 35, in open
return self._open(name, mode)
File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/storages/backends/s3boto.py", line 366, in _open
raise IOError('File does not exist: %s' % name)
IOError: File does not exist: sheets/sass/sheets.css
Complementing the answers, you could use GZIP as well when compressing:
from django.contrib.staticfiles.storage import CachedFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage
class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
def __init__(self, *args, **kwargs):
self.gzip = True
super(S3PipelineStorage, self).__init__(*args, **kwargs)
Using the settings as follows:
COMPRESS_STORAGE = STATICFILES_STORAGE = 'my.apps.main.S3PipelineStorage'
Not sure how this seemed to work for everyone else. I followed the solution above and kept getting the following error:
Traceback (most recent call last):
File "manage.py", line 24, in <module>
execute_from_command_line(sys.argv)
File "python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "python3.4/site-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "python3.4/site-packages/django/core/management/base.py", line 390, in run_from_argv
self.execute(*args, **cmd_options)
File "python3.4/site-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
collected = self.collect()
File "python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
for original_path, processed_path, processed in processor:
File "python3.4/site-packages/pipeline/storage.py", line 26, in post_process
packager.pack_stylesheets(package)
File "python3.4/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
variant=package.variant, **kwargs)
File "python3.4/site-packages/pipeline/packager.py", line 105, in pack
paths = self.compile(package.paths, force=True)
File "python3.4/site-packages/pipeline/packager.py", line 99, in compile
return self.compiler.compile(paths, force=force)
File "python3.4/site-packages/pipeline/compilers/__init__.py", line 56, in compile
return list(executor.map(_compile, paths))
File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 549, in result_iterator
yield future.result()
File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 402, in result
return self.__get_result()
File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result
raise self._exception
File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "python3.4/site-packages/pipeline/compilers/__init__.py", line 42, in _compile
outdated = compiler.is_outdated(input_path, output_path)
File "python3.4/site-packages/pipeline/compilers/__init__.py", line 85, in is_outdated
return self.storage.modified_time(infile) > self.storage.modified_time(outfile)
File "python3.4/site-packages/storages/backends/s3boto.py", line 480, in modified_time
return parse_ts(entry.last_modified)
AttributeError: 'NoneType' object has no attribute 'last_modified'
It wasn't until I came across this solution that I started to find what worked for me. Here's the storage that I ended up using that saved the file locally as well as in S3 which got me passed all errors:
from django.contrib.staticfiles.storage import ManifestFilesMixin
from django.core.files.storage import get_storage_class
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
"""Custom storage for static content."""
def __init__(self, *args, **kwargs):
super(StaticStorage, self).__init__(*args, **kwargs)
self.local_storage = get_storage_class(
'django.contrib.staticfiles.storage.StaticFilesStorage')()
def save(self, name, content):
name = super(StaticStorage, self).save(name, content)
self.local_storage._save(name, content)
return name