I am attempting to setup the django testing environment to test a script we use to load data into our data visualization web app. It is "external" to the actual django app, so it doesn't seem appropriate or clean to use the manage.py test facility.
I have followed the instructions found in the django docs to write my own test module, but when I reach a statement that actually tries to access the database I get an error saying that a table does not exist.
Is there something else I need to do to make sure that the test database is created with all of the needed tables?
For reference, here is the code from my test case:
import os
import sys
sys.path.append(os.curdir)
os.environ['DJANGO_SETTINGS_MODULE'] = 'visualization_app.settings'
from django.utils import unittest
from django.test.utils import setup_test_environment
from topic_modeling import settings
# The module to be tested
import analysis_import
setup_test_environment()
class CreateAnalysisTestCase(unittest.TestCase):
def test_analysis_creation(self):
self.assertTrue(analysis_import.create_analysis('a_name', 'a description'))
if __name__ == '__main__':
unittest.main()
The error I get looks like this:
ERROR: test_analysis_creation (__main__.CreateAnalysisTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "import_scripts/tests.py", line 20, in test_analysis_creation
self.assertTrue(analysis_import.create_analysis('a_name', 'a description'))
File "/home/dan/programmingProjects/topical_guide/import_scripts/analysis_import.py", line 117, in create_analysis
Analysis.objects.get(name=name, dataset=dataset)
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/models/query.py", line 344, in get
num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/models/query.py", line 82, in __len__
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/models/query.py", line 273, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/models/sql/compiler.py", line 680, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/models/sql/compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/backends/util.py", line 34, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/Django-1.3-py2.7.egg/django/db/backends/sqlite3/base.py", line 234, in execute
return Database.Cursor.execute(self, query, params)
DatabaseError: no such table: visualize_analysis
Strictly speaking, test databases are intended to be created, used to run unit tests and then destroyed. They're not really a place to "test" a data loading script. Why not just use your standard dev database? There shouldn't be anything there that you can't get back, have a backup for, etc., and it's pretty much designed for this sort of destructive behavior.
As far as I can see, the module you are testing still uses django's ORM system to read the database, so there is anything wrong about using manage.py to run your test. You can use
./manage.py test analysis_import
to only run test codes which reside in analysis_import module. In this way, you can still run only the tests for this module, but avoid the trouble of writing your own test module.
Then let us know if you still have the same problem.
Related
I have been trying to set up Django with a cookiecutter-django project
with cookiecutter https://github.com/pydanny/cookiecutter-django.git
and I get this error output when I run ./manage.py makemigrations at the end.
My python version in the virturalenv is Python 3.5.2
This is the tail of console output and not from a file.
Stack overflow doesn't allow me to post all of the long error output without more of other text.
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/db/models/base.py", line 156, in __new__
new_class.add_to_class(obj_name, obj)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/db/models/base.py", line 321, in add_to_class
value.contribute_to_class(cls, name)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/db/models/fields/related.py", line 1588, in contribute_to_class
self.remote_field.through = create_many_to_many_intermediary_model(self, cls)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/db/models/fields/related.py", line 1057, in create_many_to_many_intermediary_model
'verbose_name': _('%(from)s-%(to)s relationship') % {'from': from_, 'to': to},
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/functional.py", line 192, in __mod__
return str(self) % rhs
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/functional.py", line 156, in __text_cast
return func(*self.__args, **self.__kw)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/translation/__init__.py", line 79, in gettext
return _trans.gettext(message)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/translation/trans_real.py", line 285, in gettext
_default = _default or translation(settings.LANGUAGE_CODE)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/translation/trans_real.py", line 198, in translation
_translations[language] = DjangoTranslation(language)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/translation/trans_real.py", line 99, in __init__
self._add_local_translations()
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/translation/trans_real.py", line 151, in _add_local_translations
translation = self._new_gnu_trans(localedir)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/site-packages/django/utils/translation/trans_real.py", line 123, in _new_gnu_trans
fallback=use_null_fallback,
File "/usr/lib/python3.5/gettext.py", line 428, in translation
mofiles = find(domain, localedir, languages, all=True)
File "/usr/lib/python3.5/gettext.py", line 396, in find
mofile = os.path.join(localedir, lang, 'LC_MESSAGES', '%s.mo' % domain)
File "/root/painlesslanguage/painlesslanguageenv/lib/python3.5/posixpath.py", line 84, in join
elif not path or path.endswith(sep):
AttributeError: 'Path' object has no attribute 'endswith'
I left the TIME_ZONE cookiecutter-django setup option blank if that matters.
this seems to be a difficult error to fix or even debug.
some users have gotten this error when they have not included the static url properly in settings.py of left a trailing comma in the INSTALLED_APPS declaration.
however i got this error just when i switched a python version.
I ended up somehow solving it (at the time) by reinstalling python and relinking pyenv python to brew.
however, now its returned. there must be a way to debug this.
all the stacktrace points to files in the python install or only to the settings file. but if your settings file looks fine, its really hard to know how to fix it. After checking your settings.py INSTALLED_APPS (to check there are no trailing commas) id suggest reinstalling python and see if that helps.
if that doesnt work, then what helped me was printing out the apps being installed. to do this you will have to add print lines into the python/django code where the errors are thrown. as by default this info isnt being sent to the application layer, though it should be. these will be removed if you ever change the py install again. so its kind of a temperoray/local way to help u debug. really the framework should send the name info if it fails to import an app. by adding these lines i found what was going wrong with my import.
I am using flask migrate to for database creation & migration in flask with flask-sqlalchemy.
Everything was working fine until I changed my database user password contains '#' then it stopped working so, I updated my code based on
Writing a connection string when password contains special characters
It working for application but not for flask-migration, Its showing error while migrating
i.e on python manage.py db migrate
ValueError: invalid interpolation syntax in u'mysql://user:p%40ssword#localhost/testdb' at position 15
Here password is p#ssword and its escaped by urlquote (see above question link).
Full error stack:
Traceback (most recent call last):
File "manage.py", line 20, in <module>
manager.run()
File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/usr/local/lib/python2.7/dist-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask_migrate/__init__.py", line 177, in migrate
version_path=version_path, rev_id=rev_id)
File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 117, in revision
script_directory.run_env()
File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 407, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/local/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 22, in <module>
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 218, in set_main_option
self.set_section_option(self.config_ini_section, name, value)
File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 245, in set_section_option
self.file_config.set(section, name, value)
File "/usr/lib/python2.7/ConfigParser.py", line 752, in set
"position %d" % (value, tmp_value.find('%')))
ValueError: invalid interpolation syntax in u'mysql://user:p%40ssword#localhost/testdb' at position 15
Please help
In the migrations/env.py file, you will find the code that is responsible for this issue.
config.set_main_option('sqlalchemy.url',
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
If there are % signs in the SQLALCHEMY_DATABASE_URI, this will cause an error.
You can solve this by editing the migrations/env.py file, and changing the offending line as follows
db_url_escaped = current_app.config.get('SQLALCHEMY_DATABASE_URI').replace('%', '%%')
config.set_main_option('sqlalchemy.url', db_url_escaped)
Also see the documentation of set_main_option:
Note that this value is passed to ConfigParser.set, which supports variable interpolation using pyformat (e.g. %(some_value)s). A raw percent sign not part of an interpolation symbol must therefore be escaped, e.g. %%. The given value may refer to another value already in the file using the interpolation format.
I have a solution for this issue after experiencing it as well.
There's an issue with '%' (percent signs) in the db connection URI after you urlencode the string.
I tried substituting the percent sign with double percent signs ('%%') which gets me past the interpolation error. However, that resulted in not being able to connect to the database because of an incorrect password.
Solution I'm going with for now is to avoid using '%' in my db password. Not a satisfactory solution, but will do for now. I'll make a note in "alembic"'s github of the issue. Seems using RawConfigParser in their package could help avoid this issue.
You may want to look at http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-unicode
I was having the same issue with my password and the mysql connector. using the mysql+pymysql connector allowed me to connect in application and in migration scripts.
I need to create a custom User for my app and followed exactly the example from the documentation with the AUTH_USER_MODEL = 'core.MyUser' in my settings.py. However, when I make a new database, delete all the migrations folders and run the python manage.py migrate again, it gives me the exception like this
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/core/management/commands/makemigrations.py", line 173, in handle
migration_name=self.migration_name,
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/migrations/autodetector.py", line 47, in changes
changes = self._detect_changes(convert_apps, graph)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/migrations/autodetector.py", line 132, in _detect_changes
self.old_apps = self.from_state.concrete_apps
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/migrations/state.py", line 180, in concrete_apps
self.apps = StateApps(self.real_apps, self.models, ignore_swappable=True)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/migrations/state.py", line 242, in __init__
self.render_multiple(list(models.values()) + self.real_models)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/migrations/state.py", line 277, in render_multiple
model.render(self)
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/migrations/state.py", line 559, in render
body,
File "/Users/bubuzzz/Projects/python/apps/lib/python2.7/site-packages/django/db/models/base.py", line 153, in __new__
raise TypeError("%s cannot proxy the swapped model '%s'." % (name, base_meta.swapped))
TypeError: Customer cannot proxy the swapped model 'core.MyUser'.
I am not sure why there is a migrations script for the customer there, since in my app, I used to have the Customer model as well, though I deleted it already.
Then, I created a new django project to test and try to run the migration. Surprisingly, I also see those customer migration steps, but it run successfully.
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_customer... OK
Applying auth.0010_delete_customer... OK
Applying sessions.0001_initial... OK
In short, how can I create the custom User in Django 1.10 ? Example code can be viewed in here https://github.com/bubuzzz/django-customer-swap-exception
You should not delete your migration folder. If you do that, django won't make migrations for you. Create migrations folder in your core app, create an empty __init__.py file inside it, remove your db.sqlite3 file, run ./manage.py makemigrations, and then migrate should work perfectly.
Mehdi Pourfar's answer is correct.If you want to know more details
By running makemigrations, you’re telling Django that you’ve made some changes to your models (in this case, you’ve made new ones) and that you’d like the changes to be stored as a migration.
Migrations are how Django stores changes to your models (and thus your database schema) - they’re just files on disk. You can read the migration for your new model if you like; it’s the file polls/migrations/0001_initial.py. Don’t worry, you’re not expected to read them every time Django makes one, but they’re designed to be human-editable in case you want to manually tweak how Django changes things.
Tell django which app you want to make migrations all fix your problem.It will auto create a folder call migrations keep your model's record.
python manage.py makemigrations core
I'm a newbie to MySQL, and haven't used Django for anything in production. I've been doing development under Windows 7 with SQLite, and am trying to move the code to a Linux server running MySQL.
When I ran syncdb, it created the first three tables then threw an error:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/core/management/base.py", line 415, in handle
return self.handle_noargs(**options)
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/core/management/commands/syncdb.py", line 96, in handle_noargs
sql, references = connection.creation.sql_create_model(model, self.style, seen_models)
File "/home/gsdemo01/project_file/lib/python3.3/site-packages/django/db/backends/creation.py", line 83, in sql_create_model
model, f, known_models, style)
TypeError: sql_for_inline_foreign_key_references() takes 4 positional arguments but 5 were given
The model immediately after the last one created is a Metaclass, followed by a class that inherits from it:
class Location(models.Model):
# con = models.ForeignKey(Convention)
name = models.CharField(max_length=40)
scheduler = models.ForeignKey(GCUser, blank=True, null=True)
class Meta:
abstract = True
def __str__(self):
return self.name
class Room(Location):
tag = models.CharField(max_length=10)
capacity = models.PositiveSmallIntegerField(default=100)
This works fine with SQLite, so it seems like it has to be either a problem with the python connector package or with MySQL itself.
Thanks in advance for any help.
Update:
Through trial and error I've determined that the issue is definitely with the Foreign Key field. If I comment that out the table is created correctly. As far as I can tell the modification suggested by Yogesh matches what we already have in the code, so if anyone has any other suggestions I'd love to hear them.
If you using these virsions Python3.3 Django1.6 use MySql official middleware mysql-connector-python-1.1.4-py3.3.
The solution:
find: creation.py within your site-packages and Modification method: sql_for_inline_foreign_key_references is: you can only modify the method parameters
def sql_for_inline_foreign_key_references (self, model, field, known_models, style):
This is not a good solution,this is alternate for remove error.
I'm posting the solution here for the benefit of future users who run into this. It took many hours on the part of my partner to ferret this out.
The simple part of the solution is to edit creation.py to change
def sql_for_inline_foreign_key_references (self, model, field, known_models, style):
to
def sql_for_inline_foreign_key_references (self, *args):
There's a routine with the same name and almost the same syntax in site-packages/django/db/backends/creation.py, but you want the one in site-packages/django/db/backends/mysql/creation.py.
You're not done yet, though, because the module gets copied to site-packages/mysql/connector/django/creation.py, and that doesn't happen automatically on edit, so you'll need to copy it over manually. I deleted the .pyc file for good measure, but that "shouldn't" have been necessary.
Of course, none of this should have been necessary...
For Mac users using MacOSX 10.9 (aka Mavericks) or up, this solution works perfectly, but be sure you're going to the right place if you're using python3 to edit the creation.py file from the mysql.connector:
/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/mysql/connector/django/creation.py
FYI:
Environment: Python3 / Mysql-connector-1.1.4 / Django-1.6.1
My web2py application returned me an error today, which is quite odd.
Traceback (most recent call last):
File "/var/www/web2py/gluon/restricted.py", line 212, in restricted
exec ccode in environment
File "/var/www/web2py/applications/1MedCloud/controllers/default.py", line 475, in <module>
File "/var/www/web2py/gluon/globals.py", line 194, in <lambda>
self._caller = lambda f: f()
File "/var/www/web2py/applications/1MedCloud/controllers/default.py", line 63, in patient_register
rows = db(db.patientaccount.email==email).select()
File "/var/www/web2py/gluon/dal.py", line 7837, in __getattr__
return ogetattr(self, key)
AttributeError: 'DAL' object has no attribute 'patientaccount'
I am using Mysql as the database, and the table 'patientaccount' does exist. There is no connection issue as I can create tables but not fetch them from the server.
I have been using the very same code to do the db thing, here is my code
db = DAL('mysql://###:$$$#^^^^^^:3306/account_info', pool_size=0)
rows = db(db.patientaccount.email==email).select()
I did not change any code in my default.py file, but accidentally deleted some files inside "database" folder in my application. But I doubt if that could result the error, since the module is fetching tables on the server rather than using local files.
Please help! Thanks in advance!
The DAL does not inspect the MySQL database to discover its tables and fields. You must define the data models explicitly. So, somewhere in your code, you must do:
db.define_table('patientaccount',
Field('email'),
...)
That will define the db.patientaccount table so the DAL knows it exists and what fields it includes.