Working on a fairly large/complex Django project with a team, we occasionally see runserver crash with ValueError: embedded null byte. We restart runserver and it's fine - either for a few minutes or a few days. We can detect no pattern to what causes the crashes (seems totally random). Fortunately it's only happening in local development, not on our servers, but I'm worried that it will bite us down the road.
The stack trace below does not point to any location in our code -- seems to come either from Django or the virtualenv itself.
Using Django 1.9.8, Python 3.5.0, on El Capitan.
I can't see any way to debug this. Theories?
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 54, in execute
super(Command, self).execute(*args, **options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 93, in handle
self.run(**options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 102, in run
autoreload.main(self.inner_run, None, options)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 333, in main
reloader(wrapped_main_func, args, kwargs)
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 299, in python_reloader
reloader_thread()
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 275, in reloader_thread
change = fn()
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 204, in code_changed
for filename in gen_filenames():
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 114, in gen_filenames
basedirs = [os.path.abspath(basedir) for basedir in basedirs
File "/path/to/virtualenvs/ourproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 115, in <listcomp>
if os.path.isdir(basedir)]
File "/path/to/virtualenvs/ourproj/bin/../lib/python3.5/genericpath.py", line 42, in isdir
st = os.stat(s)
ValueError: embedded null byte
One of the AppConfig objects has null byte in its path attribute.
One of the LOCALE_PATHS has null byte.
One of the files is in the "wrong" encoding, so Django treats something has null byte (e. g. AppConfig.path).
One of the files or directories in project directory has null byte (\x00) in its name.
Other reason.
autoreload.py lines related to this issue. os.path.isdir() raises ValueError: embedded null byte if its argument has null byte, e. g. os.path.isdir('foo\x00bar').
You can try to edit autoreload.py, comment out these lines temporarily:
basedirs = [os.path.abspath(basedir) for basedir in basedirs
if os.path.isdir(basedir)]
and add this:
temp_basedirs = []
for basedir in basedirs:
try:
if os.path.isdir(basedir):
temp_basedirs.append(os.path.abspath(basedir))
except ValueError:
print(basedir)
raise
basedirs = temp_basedirs
Related
So I've been working in a project for a while, and haven't really changed the models at all, and therefore haven't done any migrations. Now I needed to add two new fields and delete another one, which should be normally fine. I'm using django-tenants, so my command to run the migrations is ./manage.py migrate_schemas.
Now, whenever I run that, I get the error KeyError: "prune" (the full traceback is below) in what seems to be internal code of Django.
Now, afterwards I tried reverting my changes, so no new migration and running the comnad again, and got the same error. I also thought that maybe the database had gotten "dirty" at some point, so I tried migrating a clean database from scratch, with the same result. The migrations don't even get to start.
Has anyone ever encountered something similar?
The full traceback (I have simplified the paths)
[standard:public] === Starting migration
Traceback (most recent call last):
File ":directory/./manage.py", line 22, in <module>
main()
File ":directory/./manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "$venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "$/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "$venv/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "$venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
File "$venv/lib/python3.9/site-packages/django_tenants/management/commands/migrate_schemas.py", line 52, in handle
executor.run_migrations(tenants=[self.PUBLIC_SCHEMA_NAME])
File "$venv/lib/python3.9/site-packages/django_tenants/migration_executors/standard.py", line 11, in run_migrations
run_migrations(self.args, self.options, self.codename, self.PUBLIC_SCHEMA_NAME)
File "$venv/lib/python3.9/site-packages/django_tenants/migration_executors/base.py", line 53, in run_migrations
MigrateCommand(stdout=stdout, stderr=stderr).execute(*args, **options)
File "$venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
File "$venv/lib/python3.9/site-packages/django/core/management/base.py", line 96, in wrapped
res = handle_func(*args, **kwargs)
File "$venv/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 188, in handle
if options["prune"]:
KeyError: 'prune'
It seems like version incompatibility.
prune option added to Django couple months ago (Jan 22, 2022)
If you want to use newer Django you have to patch django-tenants manually and add --prune argument
def add_arguments(self, parser):
parser.add_argument(
'--prune', action='store_true', dest='prune',
help='Delete nonexistent migrations from the django_migrations table.',
)
PS I couldn't find issue related to prune, so you may create new one ;)
So, I`m comparing some python test frameworks and came across behave. Thought it was interesting and worth a test drive.
Followed the steps on the tutorial, available at:
https://behave.readthedocs.io/en/stable/tutorial.html
When I ran the behave command on Powershell (Win10 and Python 2.7.10), I got the following error:
Exception TypeError: compile() expected string without null bytes
Traceback (most recent call last):
File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\Python27\lib\site-packages\behave\__main__.py", line 187, in <module>
sys.exit(main())
File "C:\Python27\lib\site-packages\behave\__main__.py", line 183, in main
return run_behave(config)
File "C:\Python27\lib\site-packages\behave\__main__.py", line 127, in run_behave
failed = runner.run()
File "C:\Python27\lib\site-packages\behave\runner.py", line 804, in run
return self.run_with_paths()
File "C:\Python27\lib\site-packages\behave\runner.py", line 809, in run_with_paths
self.load_step_definitions()
File "C:\Python27\lib\site-packages\behave\runner.py", line 796, in load_step_definitions
load_step_modules(step_paths)
File "C:\Python27\lib\site-packages\behave\runner_util.py", line 412, in load_step_modules
exec_file(os.path.join(path, name), step_module_globals)
File "C:\Python27\lib\site-packages\behave\runner_util.py", line 385, in exec_file
code = compile(f.read(), filename2, "exec", dont_inherit=True)
TypeError: compile() expected string without null bytes
Has anyone encountered this error while trying to run behave? (Found some threads online related mainly to flask issues but I couldn't solve the problem)
Answering my own question here.
It was an encoding problem.
Sublime was saving my files with an encoding different from UTF-8.
File -> Save with Encoding -> UTF-8 did the trick.
I'm getting an error when I try to encrypt the body of a SOAP XML message using https://pypi.org/project/dm.xmlsec.binding/.
Traceback (most recent call last):
File "./manage.py", line 14, in <module>
execute_from_command_line(sys.argv)
File "/Users/romeroqj/.virtualenv/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/Users/romeroqj/.virtualenv/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/romeroqj/.virtualenv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/romeroqj/.virtualenv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/Users/romeroqj/Desktop/management/commands/encrypt.py", line 63, in handle
encrypted_xml = enc_ctx.encryptXml(enc_data, doc.getroot())
File "_xmlsec.pyx", line 555, in dm.xmlsec.binding._xmlsec.EncCtx.encryptXml (src/_xmlsec.c:7336)
File "_xmlsec.pyx", line 637, in dm.xmlsec.binding._xmlsec.lxml_safe_dealloc (src/_xmlsec.c:8132)
File "src/lxml/public-api.pxi", line 29, in lxml.etree.elementFactory
File "src/lxml/etree.pyx", line 1607, in lxml.etree._elementFactory
File "src/lxml/classlookup.pxi", line 403, in lxml.etree._parser_class_lookup
File "src/lxml/nsclasses.pxi", line 174, in lxml.etree._find_nselement_class
File "src/lxml/classlookup.pxi", line 257, in lxml.etree._callLookupFallback
File "src/lxml/classlookup.pxi", line 336, in lxml.etree._lookupDefaultElementClass
AssertionError: Unknown node type: 3
The error occurs when I try to encrypt the content of the body (xmlsec.TypeEncContent), but it works when I try to encrypt the body element (xmlsec.TypeEncElement). Here's the code:
import dm.xmlsec.binding as xmlsec
from dm.xmlsec.binding.tmpl import EncData, fromstring
key_mngr = xmlsec.KeysMngr()
key = xmlsec.Key.load('cert.pem', xmlsec.KeyDataFormatCertPem)
key.name = 'cert.pem'
key_mngr.addKey(key)
enc_data = EncData(
xmlsec.TransformAes128Cbc, type=xmlsec.TypeEncContent
)
enc_data.ensureCipherValue() # target for encryption result
enc_ctx = xmlsec.EncCtx(key_mngr)
enc_ctx.encKey = xmlsec.Key.generate(
xmlsec.KeyDataAes, 192, xmlsec.KeyDataTypeSession
)
doc = fromstring(xml_data)
body = doc.find('soap:Body', namespaces=NAMESPACES) # NAMESPACES is redacted for brevity
encrypted_xml = enc_ctx.encryptXml(enc_data, doc.getroot())
The XML data looks something like this:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
...
</soap:Header>
<soap:Body Id="Body">
...
</soap:Body>
</soap:Envelope>
What could be wrong here?
I'm not sure if this is relevant or related, but I tried to do the same using a different library https://github.com/mehcode/python-xmlsec/issues/83 and I got a Segmentation fault error, however, it worked when I tried to encrypt the element. This pattern makes me believe there's something wrong in this approach.
Any hints are appreciated. Thanks in advance!
I'm working with Django 1.9 in Microsoft Visual Studio. Also I'm using a virtual env, where all the needed python packages are installed. When I try to compile messages in order to add translation function to the site, the following error is raised:
ValueError: plural forms expression could be dangerous
Update 1. Stacktrace:
Internal Server Error: /
Traceback (most recent call last):
File "C:\path-to-project\env\lib\site-packages\django\core\handlers\base.py", line 123, in get_response
response = middleware_method(request)
File "C:\path-to-project\env\lib\site-packages\django\middleware\locale.py", line 26, in process_request
translation.activate(language)
File "C:\path-to-project\env\lib\site-packages\django\utils\translation\__init__.py", line 154, in activate
return _trans.activate(language)
File "C:\path-to-project\env\lib\site-packages\django\utils\translation\trans_real.py", line 216, in activate
_active.value = translation(language)
File "C:\path-to-project\env\lib\site-packages\django\utils\translation\trans_real.py", line 205, in translation
_translations[language] = DjangoTranslation(language)
File "C:\path-to-project\env\lib\site-packages\django\utils\translation\trans_real.py", line 114, in __init__
self._add_installed_apps_translations()
File "C:\path-to-project\env\lib\site-packages\django\utils\translation\trans_real.py", line 160, in _add_installed_apps_translations
translation = self._new_gnu_trans(localedir)
File "C:\path-to-project\env\lib\site-packages\django\utils\translation\trans_real.py", line 140, in _new_gnu_trans
fallback=use_null_fallback)
File "C:\Python27\Lib\gettext.py", line 481, in translation
t = _translations.setdefault(key, class_(fp))
File "C:\Python27\Lib\gettext.py", line 182, in __init__
self._parse(fp)
File "C:\Python27\Lib\gettext.py", line 318, in _parse
self.plural = c2py(plural)
File "C:\Python27\Lib\gettext.py", line 95, in c2py
raise ValueError, 'plural forms expression could be dangerous'
ValueError: plural forms expression could be dangerous
I've read about that (plural forms expression could be dangerous and Django: gettext raises ValueError: 'plural forms expression could be dangerous'), but the only solution I found is to remove lines referred to plural translations, for example:
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
If I remove those lines in my translations files, it still raises the same error, because it takes into account other files located at the virtual env.
Update 2: If I run the server from the command line, using
python manage.py runserver
there's no problem, everything is going right. However, when I run the project from Visual Studio, the error is raised.
Is there any way to solve this problem?
Thanks in advance.
I just generated the migration scripts through ./manage.py schemamigration --auto and ran it. I get the following error. I am stumped as to what it could mean. I have been using SET_NULL for a while now. So this is something new that didn't occur earlier. Any idea what could be wrong?
Traceback (most recent call last):
File "./manage.py", line 16, in
execute_from_command_line(sys.argv)
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/django/core/management/init.py", line 399, in execute_from_command_line
utility.execute()
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/django/core/management/init.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.dict)
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/south/management/commands/schemamigration.py", line 111, in handle
old_orm = last_migration.orm(),
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/south/utils/init.py", line 62, in method
value = function(self)
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/south/migration/base.py", line 432, in orm
return FakeORM(self.migration_class(), self.app_label())
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/south/orm.py", line 48, in FakeORM
_orm_cache[args] = _FakeORM(*args)
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/south/orm.py", line 134, in init
self.retry_failed_fields()
File "/home/vivekv/.environments/fantain/local/lib/python2.7/site-packages/south/orm.py", line 377, in retry_failed_fields
fname, modelname, e
ValueError: Cannot successfully create field 'winner' for model 'match': 'module' object has no attribute 'SET_NULL'.
OK This is not a valid question. I am embarrassed to admit I made a small tweak on the migration script that caused the problem. Please ignore this question - seems like I dont have a way to delete a question I had asked!
I wrote the solution here
Short answer: use models.SET_NULL instead of "SET_NULL".
You need to replace this in all migrations which have b"SET_NULL"