I want build a blog with Python 2.7 and Django 1.7.8.
When I use Django,I keep getting an error: UnicodeDecodeError.
The relevant code is:
#coding: utf-8
from django.db import models
class Entry(models.Model):
title = models.CharField(max_length=200)
body = models.TextField()
publish = models.BooleanField(default=False)
And ./manage.py makemigrations get an error:
Migrations for 'blog':
0001_initial.py:
- Create model Entry
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/core/management/commands/makemigrations.py", line 124, in handle
self.write_migration_files(changes)
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/core/management/commands/makemigrations.py", line 143, in write_migration_files
migrations_directory = os.path.dirname(writer.path)
File "/usr/local/lib/python2.7/dist-packages/Django-1.7.8-py2.7.egg/django/db/migrations/writer.py", line 222, in path
return os.path.join(basedir, self.filename)
File "/usr/lib/python2.7/posixpath.py", line 80, in join
path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 10: ordinal not in range(128)
I don't see what's going wrong, any idea? I already add coding:utf-8 in top of code
do you have data in the models already?
if so, i am guessing you have some bad characters in there.
do you have a unicode or str method defined on the models?
if so, i suggest using them in a try statement of sorts like this:
def __str__():
try:
return "%s" % self.title
except:
return "%s" % self.pk
when you see only the PK, you will know that your 'title' field has bad data.
extend this to include whichever fields you want to display, not just title.
it looks like it fails when joining the path?
maybe you have folder names that can't be converted to ascii?
also , consider editing the file that is having the error? maybe add a print statement just before the line that fails that shows what is trying to be joined?
edit the file mentioned below, add the print statement to see what is trying to be joined?
File "/usr/lib/python2.7/posixpath.py", line 80, in join
path += '/' + b
Related
My task is to populate my two existing City and Province model using two json files cities.json' and 'provinces.json.
Data in these files are as below.
provinces.json:
[
{ "model": "salesnetwork.provinces",
"id": "1",
"name": "EA"
}
]
cities.json:
[
{ "model": "salesnetwork.cities",
"id": "1",
"province_id": "1",
"name": "Tabriz"
}
]
Now I'm trying to poplulate my two models with these data. My models are as below.
class Provinces(models.Model):
name = models.CharField(max_length=255, db_collation="utf8mb3_unicode_ci")
class Meta:
db_table = "provinces"
class Cities(models.Model):
province = models.ForeignKey("Provinces", models.DO_NOTHING)
name = models.CharField(max_length=255, db_collation="utf8mb3_unicode_ci")
class Meta:
db_table = "cities"
(My bad for naming my model Cities instead of City and Provinces instead of 'Province'; I've created these models via inspectdb command and since I didn't know if changing these model names could cause problems or not, I decided to leave the names as they were).
When I try using command py .\manage.py loaddata provinces.json I get the following error:
Traceback (most recent call last):
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\serializers\json.py", line 69, in Deserializer
objects = json.loads(stream_or_string)
File "C:\Users\Vahid Moradi\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 335, in loads
raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\manage.py", line 22, in <module>
main()
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\__init__.py", line 446, in execute_from_command_line
utility.execute()
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\base.py", line 448, in execute
output = self.handle(*args, **options)
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\commands\loaddata.py", line 102, in handle
self.loaddata(fixture_labels)
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\commands\loaddata.py", line 163, in loaddata
self.load_label(fixture_label)
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\management\commands\loaddata.py", line 251, in load_label
for obj in objects:
File "D:\Projects\Navid Motor\Website\Django\NavidMotor.com\.venv\lib\site-packages\django\core\serializers\json.py", line 74, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture 'D:\Projects\Navid Motor\Website\Django\NavidMotor.com\provinces.json'::
and by the looks of it, this time the problem is json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0), which I have no idea how to resolve and didn't find any instruction on how to solve this issue.
I did study the following questions, but all were for problems relating json library inside .py files, and since my problem occurs in command line using manage.py commands, I couldn't apply those solutions for my problem.
Possible simillar questions, but could not use the answers given:
First
Second
Third
Try to paste json into new file in vscode and save it.
Errorjson.decoder.JSONDecodeError: Unexpected UTF-8 BOM because your json files have UTF8-BOM encoding:
https://www.coderedcorp.com/blog/how-to-dump-your-django-database-and-load-it-into-/
I've got a problem with the command loaddata of my Django app.
When I run this command, this UnicodeDecodeError appears because of the "é" that are in my database table.
i've try with many type of database on pgAdmin with different encode type like "UTF-8", "latin-1" or "win1252" or different "character type" or "collation" settings like "C" or "french_switzerland.1252" but every time there is this error.
Here is more information about my work space.
Let me know if you need more informations, I don't give you too much, I don't want to give useless information.
I know there are other questions which look similar to mine but I don't understand the answers. There are explanations about why there is the error but not about how to solve it.
I am on Windows 10
I'm using Pycharm 2020.3.2
Pgadmin v4
the complete error :
(venv) C:\Users\Mathias\PycharmProjects\yufindProject>python manage.py loaddata ask/dumps/ask.json
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
utility.execute()
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\__init__.py", line 395, in execut
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\base.py", line 371, in execute
output = self.handle(*args, **options)
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
self.load_label(fixture_label)
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 172, in load_label
for obj in objects:
File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\serializers\json.py", line 67, in Deserializer
stream_or_string = stream_or_string.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 460: invalid continuation byte
Found the solution in Django dumpdata fails on special characters
To save json data in django the TextIOWrapper is used:
The default encoding is now locale.getpreferredencoding(False) (...)
In documentation of locale.getpreferredencoding fuction we can
read:
Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different
systems, and might not be available programmatically on some systems,
so this function only returns a guess.
Here I found "hacky" but working method to overwrite these
settings:
In file settings.py of your django project add these lines:
import _locale
_locale._getdefaultlocale = (lambda *args: ['en_US', 'utf8'])
Working on a fairly large/complex Django project with a team, we occasionally see runserver crash with ValueError: embedded null byte. We restart runserver and it's fine - either for a few minutes or a few days. We can detect no pattern to what causes the crashes (seems totally random). Fortunately it's only happening in local development, not on our servers, but I'm worried that it will bite us down the road.
The stack trace below does not point to any location in our code -- seems to come either from Django or the virtualenv itself.
Using Django 1.9.8, Python 3.5.0, on El Capitan.
I can't see any way to debug this. Theories?
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 54, in execute
super(Command, self).execute(*args, **options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 93, in handle
self.run(**options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 102, in run
autoreload.main(self.inner_run, None, options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 333, in main
reloader(wrapped_main_func, args, kwargs)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 299, in python_reloader
reloader_thread()
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 275, in reloader_thread
change = fn()
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 204, in code_changed
for filename in gen_filenames():
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 114, in gen_filenames
basedirs = [os.path.abspath(basedir) for basedir in basedirs
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 115, in <listcomp>
if os.path.isdir(basedir)]
File "/path/to/virtualenvs/ourproj/bin/../lib/python3.5/genericpath.py", line 42, in isdir
st = os.stat(s)
ValueError: embedded null byte
One of the AppConfig objects has null byte in its path attribute.
One of the LOCALE_PATHS has null byte.
One of the files is in the "wrong" encoding, so Django treats something has null byte (e. g. AppConfig.path).
One of the files or directories in project directory has null byte (\x00) in its name.
Other reason.
autoreload.py lines related to this issue. os.path.isdir() raises ValueError: embedded null byte if its argument has null byte, e. g. os.path.isdir('foo\x00bar').
You can try to edit autoreload.py, comment out these lines temporarily:
basedirs = [os.path.abspath(basedir) for basedir in basedirs
if os.path.isdir(basedir)]
and add this:
temp_basedirs = []
for basedir in basedirs:
try:
if os.path.isdir(basedir):
temp_basedirs.append(os.path.abspath(basedir))
except ValueError:
print(basedir)
raise
basedirs = temp_basedirs
I am building a website with django/python. The website database I am building contains books, and I am trying to relate Character objects to Event objects by defining their relationships in the fixture. I load my fixtures from .json files using
loaddata fixtures <file>
This works for my models that have no relationships, but when I try to load a fixture that contains many-to-many relationships (characters appear in many events, events contain many characters) I get the following :
ValueError: Problem installing fixture: < file path >: "< Character: Lanoree Brock > " needs to have a value for field "character" before this many-to-many relationship can be used.
There is no field "character" in my model for Character:
class Character(models.Model):
id = models.IntegerField(primary_key = True)
name = models.CharField(max_length = ml)
bio = models.TextField()
event = models.ManyToManyField(Event)
def __str__(self):
return self.name
class Meta:
ordering = ('name',)
The .json file for my Character fixture looks like this:
[{"model": "library.Character", "id": 1,
"fields": {"name": "Lanoree Brock", "bio": "He lived", "event": [101, 102, ...]}}
... ]
So the error occurs at the first Character. My guess as to the problem is that django tries to add the relationship to the < Character: Lanoree Brock > object before it saves the object, but I do not understand why it is doing that or how to get around it.
Is there a way to structure the fixture to ensure that when it is loaded each object is created/saved before the code tries to define its relationships?
I am new to django, JSON syntax, and web dev in general, and I feel like there's something simple I'm not doing here -- if fixtures cannot handle many-to-many relationships, that seems like a huge oversight in their functionality.
Any help would be appreciated, thank you!
Edit: The full error log:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\__init__.py", line 353, in execute_from_command_line
utility.execute()
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\base.py", line 399, in execute
output = self.handle(*args, **options)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\commands\loaddata.py", line 60, in handle
self.loaddata(fixture_labels)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\commands\loaddata.py", line 100, in loaddata
self.load_label(fixture_label)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\managemen
t\commands\loaddata.py", line 158, in load_label
obj.save(using=self.using)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\core\serialize
rs\base.py", line 204, in save
setattr(self.object, accessor_name, object_list)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\db\models\fiel
ds\related_descriptors.py", line 480, in __set__
manager = self.__get__(instance)
File "c:\Users\MetalGearSamus\Anaconda\lib\site-packages\django\db\models\fiel
ds\related_descriptors.py", line 468, in __get__
return self.related_manager_cls(instance)
File "c:Users\MetalGearSamus\Anaconda\lib\site-packages\django\db\models\fiel
ds\related_descriptors.py", line 751, in __init__
(instance, self.source_field_name))
ValueError: Problem installing fixture 'c:Users\MetalGearSamus\Personal\Legends\website\library\fixtures\database.json': "<Character: Lanoree Brock>" needs to
have a value for field "character" before this many-to-many relationship can be
used.
I have the following situation. Project files was stored in ~/django-apps/app-old/app. For some reasons I move files to ~/django-apps/app . Application have some images, stored with use of ImageField. In database images have paths like so:
~/django-apps/app-old/app/media/images/blabla.jpeg
So, I need to fix this paths in database to look like this:
~django-apps/app/media/images/blabla.jpeg
I try to write management command for to do this:
from django.core.management.base import BaseCommand, CommandError
from books.models import Book
import string
class Command(BaseCommand):
help = ''
def handle(self, *args, **options):
books = Book.objects.all()
total = len(books)
curr = 1
for book in books:
print "%d/%d" % (curr, total)
if book.cover_url != "":
book.cover_url.path = string.replace(book.cover_url.path, "app-old/", "")
book.save()
curr+=1
By using this command I get following error:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 220, in execute
output = self.handle(*args, **options)
File "/var/www/dizpers/data/django-apps/app/books/management/commands/fix-covers-path.py", line 23, in handle
book.cover_url.path = string.replace(book.cover_url.path, "bparser-old/", "")
File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/files.py", line 65, in _get_path
return self.storage.path(self.name)
File "/usr/local/lib/python2.6/dist-packages/django/core/files/storage.py", line 234, in path
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
django.core.exceptions.SuspiciousOperation: Attempted access to '/var/www/dizpers/data/django-apps/app-old/app/media/covers/a3d9545d3a17bb68a91749019c95357d.jpeg' denied.
Why I get this error message? How I can fix image's path?
UPD1
My model contain ImageField like this:
cover_url = models.ImageField(upload_to=os.path.join(MEDIA_ROOT, "covers"), null=True, default=None)
UPD2
Path property is readonly
This should work
for book in books:
print "%d/%d" % (curr, total)
if book.cover_url != "":
new_path = string.replace(book.cover_url.path, "app-old/", "")
book.cover_url = new_path
book.save()
curr+=1
You get this message because the default file system storage prevents usage of absolute path for security reasons.
If you really want to use an absolute path
django-documents forces absolute paths for security reasons, this is how:
from django.core.files.storage import FileSystemStorage
fs = FileSystemStorage(location=UPLOAD_TO)
class Document(models.Model):
file = models.FileField(storage=fs, upload_to=UPLOAD_TO)
Where UPLOAD_TO can be an absolute path.
-- UPDATE IN REPLY TO QUESTION UPDATE --
If you don't need such security
Then you should read the documentation on FileField:
FileField.upload_to
A local filesystem path that will be appended to your MEDIA_ROOT setting to determine the value of the url attribute.
In your case, upload_to='covers' is sufficient.
Note that relative paths should be stored in the database.