I have two databases defined in my setting.py file.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'monitoring',
'USER': 'root',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '',
},
'source' :{
'ENGINE': 'django.db.backends.mysql',
'NAME': 'source_db',
'USER': '*****',
'PASSWORD': '*****',
'HOST': '*****',
'PORT': '****',
}
}
I need to access some tables in source_db which django is not allowing to do so unless I migrate the db. So, once we run command python manage.py migrate --database=source , Django is creating some tables in server db. Since we are not allowed to create tables in server db, is there any way to stop django doing so, or any way to access tables without migrating the db?
This is the list of tables which we don't want to create.
+--------------------------------+
| Tables_in_source_db |
+--------------------------------+
| auth_group |
| auth_group_permissions |
| auth_permission |
| auth_user |
| auth_user_groups |
| auth_user_user_permissions |
| dashboard_monitoring_features |
| dashboard_monitoring_modelinfo |
| dashboard_monitoring_product |
| django_admin_log |
| django_content_type |
| django_migrations |
| django_session |
+--------------------------------+
If you want to migrate / create tables in the default database and not source database, then you have to define app and database when you run migrations. Like:
python manage.py migrate dashboard --database=default
This will run migration in the dashboard app and create tables in the default database.
Next thing you want to do is set your source models to non-managed. You do by specifying managed = False in the meta class of the model:
class YourModel(models.Model):
... your fields here ...
class Meta:
managed = False
From the documentation:
If False, no database table creation or deletion operations will be
performed for this model. This is useful if the model represents an
existing table or a database view that has been created by some other
means.
If you don't want particular tables, then remove
'django.contrib.auth`,
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.admin',
and dashboard app
from your INSTALLED_APPS
Probably nobody was clear about what the author of the question was trying to do. So let me start with the context.
Context
By default, Django uses a database named default. Let's say I have a CustomModel in a CustomApp for which the table should be created in a new database called source_db.
Thus we have two databases: 1)default and 2)source_db.
We want the default tables like auth_group required by apps like django.contrib.auth to be created only in default. The source_db should have and only have tables from CustomApp. How do we do this?
Solution 1: I recommend this method
First, refer to Multiple Databases, the official documentation to understand the routing basics. Give special attention to allow_migrate(). Let me insist, understanding of this documentation is at most important to understand the rest of this solution.
Let's begin:
In your CustomModel, set managed = True in the meta data:
class CustomModel(models.Model):
# your fields here ...
class Meta:
managed = True
This specifies that the table creation and alteration for this model should be taken care of by Django.
Write your CustomDBRouter for tables in CustomApp. (Refer to Multiple Databases for this). Don't forget to define allow_migrate().
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""
Make sure the 'CustomApp' only appear in the 'source_db' database.
"""
if app_label == 'CustomApp':
# if app=CustomApp and db=source_db, migrate
return db == 'source_db'
return None # No suggestions, ask the next router in the list
Go to your settings.py and split the list INSTALLED_APPS as follows:
MY_APPS = [ # List of apps defined by you
'customApp'
]
INSTALLED_APPS = [ # Full list of apps
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles'
] + MY_APPS
Now add a new router to project-app(default-app) folder (The folder where you have your settings.py)
This router should look like:
from project.settings import MY_APPS
class DefaultDBRouter(object):
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""
If an app is not defined by me, migrate only if database is 'default'
"""
if app_label not in MY_APPS:
return db == 'default'
return False
Now when adding DefaultDBRouter to DATABASE_ROUTERS in settings.py, make sure that this is the last router in the list. This is at most important since the routers are processed in the order listed.
DATABASE_ROUTERS = ['customApp.dbRouter.CustomDBRouter', 'project.dbRouter.DefaultDBRouter']
Finally, you can migrate using the following commands:
python manage.py makemigrations # Make migrations for all apps
python manage.py migrate # Do migrations to 'default'
python manage.py migrate --database=source_db # Do migrations to 'source'
Solution 2: I don't recommend this
Follow step 1 and 2 of Solution 1.
Now, while doing migrations follow these steps:
python manage.py makemigrations # Make migrations for all apps
python manage.py migrate # Do migrations to `default`
python manage.py migrate customApp --database=source_db
# Do migrations to 'source_db' from `customApp` only
Since tables like auth_group do not belong to migrations of customApp, they won't be created in the last command. You can add these 3 commands to a shell script to make your job easy. Note that, as the number of apps and databases increase, this method will look dirty. On the other hand, Solution-1 provides a clean way to do this.
Was looking for another issue but stumbled upon this question. The answer is to use a database router. In the project's root_app create a file (I called mine model.py) and add the following settings:
from <app> import models as app_models
class DatabaseRouter(object):
def db_for_read(self, model, **hints):
""" Route for when Django is reading the model(s). """
if model in (
app_models.<name>,
):
return "project_db"
return "default"
def db_for_write(self, model, **hints):
""" Route for when Django is writing the model(s). """
if model in (
app_models.<name>
):
return "project_db"
return "default"
Then inside your settings.py you should have:
DATABASES = {
"default": {"ENGINE": "django.db.backends.sqlite3", "NAME": ":memory:"},
"project_db": { ... },
}
And the following line:
DATABASE_ROUTERS = ("root_app.models.DatabaseRouter",)
You should be able to generalize the above for your liking.
Related
For a project in Django I have to use two databases: default and remote. I have created routers.py and everything works fine.
There was a requirement to create a table on the remote database and I created migration, run it and the table django_migrations was created. I want to have only one table django_migrations, in the default database.
The relevant part of routers.py is here:
class MyRouter(object):
# ...
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'my_app':
return db == 'remote'
return None
I run the migration like this:
python manage.py migrate my_app --database=remote
Now when I do:
python manage.py runserver
I get the following warning:
You have 1 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): my_app.
Run 'python manage.py migrate' to apply them.
The tables for my_app are created in the remote database, and in django_migrations inside the remote database the migrations are marked as applied.
EDIT:
How to force Django to use only one table django_migrations, but still apply the migrations into different databases?
How to apply the migrations in different databases so that no warnings are raised?
Thanks to the comments on my question I did some research and came up with the following findings.
Using multiple databases results in creating a table django_migrationswhen migrations are used. There is no option to record the migrations in only one table django_migrations, as the comment from Kamil Niski explains. This is clear after reading the file django/db/migrations/recorder.py.
I will illustrate an example with a project foo and an app bar inside the project. The app bar has only one model Baz.
We create the project:
django-admin startproject foo
Now we have these contents inside the main project directory:
- foo
- manage.py
I have a habit to group all apps inside the project directory:
mkdir foo/bar
python manage.py bar foo/bar
In the file foo/settings.py we adjust the settings to use two different databases, for the purposes of this example we use sqlite3:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db1.sqlite3'),
},
'remote': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db2.sqlite3'),
}
}
Now we run the migrations:
python manage.py migrate --database=default
This runs all migrations, the part --database=default is optional, because if not specified Django uses the default database.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying sessions.0001_initial... OK
Django has applied all migrations to the default database:
1 contenttypes 0001_initial 2019-11-13 16:51:04.767382
2 auth 0001_initial 2019-11-13 16:51:04.792245
3 admin 0001_initial 2019-11-13 16:51:04.827454
4 admin 0002_logentr 2019-11-13 16:51:04.846627
5 admin 0003_logentr 2019-11-13 16:51:04.864458
6 contenttypes 0002_remove_ 2019-11-13 16:51:04.892220
7 auth 0002_alter_p 2019-11-13 16:51:04.906449
8 auth 0003_alter_u 2019-11-13 16:51:04.923902
9 auth 0004_alter_u 2019-11-13 16:51:04.941707
10 auth 0005_alter_u 2019-11-13 16:51:04.958371
11 auth 0006_require 2019-11-13 16:51:04.965527
12 auth 0007_alter_v 2019-11-13 16:51:04.981532
13 auth 0008_alter_u 2019-11-13 16:51:05.004149
14 auth 0009_alter_u 2019-11-13 16:51:05.019705
15 auth 0010_alter_g 2019-11-13 16:51:05.037023
16 auth 0011_update_ 2019-11-13 16:51:05.054449
17 sessions 0001_initial 2019-11-13 16:51:05.063868
Now we create the model Baz:
models.py:
from django.db import models
class Baz(models.Model):
name = models.CharField(max_length=255, unique=True)
register the app bar into INSTALLED_APPS (foo/settings.py) and create the migrations:
python manage.py makemigrations bar
Before we run the migrations we create routers.py inside the bar app:
class BarRouter(object):
def db_for_read(self, model, **hints):
if model._meta.app_label == 'bar':
return 'remote'
return None
def db_for_write(self, model, **hints):
if model._meta.app_label == 'bar':
return 'remote'
return None
def allow_relation(self, obj1, obj2, **hints):
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'bar':
return db == 'remote'
if db == 'remote':
return False
return None
and register it in foo/settings.py:
DATABASE_ROUTERS = ['foo.bar.routers.BarRouter']
Now the naive approach would be to run the migrations for bar into the remote database:
python manage.py migrate bar --database=remote
Operations to perform:
Apply all migrations: bar
Running migrations:
Applying bar.0001_initial... OK
The migrations has been applied to the remote database:
1 bar 0001_initial 2019-11-13 17:32:39.701784
When we run:
python manage.py runserver
the following warning will be raised:
You have 1 unapplied migration(s). Your project may not work properly
until you apply the migrations for app(s): bar.
Run 'python manage.py migrate' to apply them.
Everything seems to work fine though. However it isn't satisfying having this warning.
The proper way would be to run all migrations for each database as suggested in this answer.
It would look like this:
python manage.py migrate --database=default
python manage.py migrate --database=remote
and after creating the migrations for bar:
python manage.py migrate bar --database=default
python manage.py migrate bar --database=remote
The router will take care that the table bar_baz is created only in the remote database, but Django will mark the migrations as applied in both databases. Also the tables for auth, admin, sessions, etc. will be created only in the default database, as specified in routers.py. The table django_migrations in the remote database will have records for these migrations too.
It is a long reading, but I hope it sheds some light on this, in my opinion, not thoroughly explained issue in the official documentation.
I have a Django app that adds and displays stuff to my postgresql database online at elephantsql.com. My project file setup looks like this:
website/
website/
music/
playlist/
__pycache__/
migrations/
static/
templates/
__init__.py
admin.py
apps.py
models.py
tests.py
urls.py
views.py
.coverage
db.sqlite3
manage.py
My project works right now where when I run the server and go to /playlist/ it displays stuff correctly and connects to my postgresql database fine.
My settings.py DATABASES object looks like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'kxvmghva',
'USER': 'kxvmghva',
'PASSWORD': '--an actual password---',
'HOST': 'raja.db.elephantsql.com',
'PORT': '5432',
}
}
Now I'm trying to write test cases in my playlist/tests.py file, but when I try to run these tests I'm getting errors.
my testing file I'm trying to run /playlist/tests.py
from django.test import TestCase
from .models import plays
from django.utils import timezone
class AnimalTestCase(TestCase):
def setUp(self):
print("setup")
#Animal.objects.create(name="lion", sound="roar")
#Animal.objects.create(name="cat", sound="meow")
def test_animals_can_speak(self):
"""Animals that can speak are correctly identified"""
print("test")
#lion = Animal.objects.get(name="lion")
#cat = Animal.objects.get(name="cat")
#self.assertEqual(lion.speak(), 'The lion says "roar"')
#self.assertEqual(cat.speak(), 'The cat says "meow"')
When I run the command "python manage.py test playlist" I get these errors:
C:\Users\marti\Documents\kexp\website>python manage.py test playlist
Creating test database for alias 'default'...
C:\Python36-32\lib\site-packages\django\db\backends\postgresql\base.py:267: RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the default database instead.
RuntimeWarning
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'test_kxvmghva', or 'no' to cancel:
if I type 'yes' it leads to this error:
Destroying old test database for alias 'default'...
Got an error recreating the test database: database "test_kxvmghva" does not exist
I've been trying to solve this error by searching it online and have tried stuff like giving my user 'kxvmghva' CREATEDB permissions as well as running this line in my elephantsql db:
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO kxvmghva;
But I'm still getting these errors when trying to run the tests.py file for my playlist/ app. This is my first time setting up test cases for a postgresql database in django and I would really appreciate any help or guidance. Thanks.
I suppose the plan you are using for database is free/shared, in case which you do not have rights to create additional databases.
This is not a Django problem but restrictions of the service you are using.
As pointed in the first answer, ElephantSQL free tier (I suppose you're using this one) is shared. In that server, they have several DBs, so you have access to one and can't create another one.
I've tried to create an instance just for tests, but it also fails because "On PostgreSQL, USER will also need read access to the built-in postgres database." (from django docs).
So, my solution, and understand it is not the best of practices (because you'll have a different environment for testing) is replacing the engine for testing:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
},
}
This configuration works because (from the django docs):
When using SQLite, the tests will use an in-memory database by default (i.e., the database will be created in memory, bypassing the filesystem entirely!)
Two notes:
Use GitHub actions or a similar solution to test against PostgreSQL in each commit (this way, you reduce the danger of using different test and prod databases)
The easiest way to have a different setting config is to use separate files. I follow the logic in cookiecutter-django
I am new to Django.
I have two apps created in my project:
python3 manage.py startapp app1
python3 manage.py startapp app1
I am using Mysql as DB and I want that each app should use different schemas.
I try to follow the steps described here:
Sharing (mysql) database between apps Django with Database routers
So in settings.py I defined 2 MySql schemas and keep default schema and
add the add the DATABASE_ROUTERS.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'app1',
'app2',
]
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
},
'mydb1':{
'ENGINE': 'django.db.backends.mysql',
'NAME': 'APP1DB',
'USER': 'user1',
'PASSWORD': '****',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '3306',
},
'mydb2':{
'ENGINE': 'django.db.backends.mysql',
'NAME': 'APP2DB',
'USER': 'user1',
'PASSWORD': '****',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '3306',
},
}
DATABASE_ROUTERS = ['app1.dbRouter.App1DBRouter', 'app2.dbRouter.App2DBRouter']
Additional files:
app1/models1.py
from django.db import models
# Create your models here.
class Model1(models.Model):
name = models.CharField(max_length=100)
app2/models2.py
from django.db import models
# Create your models here.
class Model2(models.Model):
name = models.CharField(max_length=100)
And files:
app1/dbRouter.py
class App1DBRouter(object):
def db_for_read(self,model, **hints):
if model._meta.app_label == 'app1':
return 'mydb1'
return None
def db_for_write(self,model, **hints):
if model._meta.app_label == 'app1':
return 'mydb1'
return None
def allow_relation(self,obj1, obj2, **hints):
if obj1._meta.app_label == 'app1' and \
obj2._meta.app_label == 'app1':
return True
return None
def allow_syncdb(self,db, model):
if db == 'mydb1':
if model._meta.app_label == 'app1':
return True
elif model._meta.app_label == 'app1':
return False
return None
app2/dbRouter.py:
class App2DBRouter(object):
def db_for_read(self,model, **hints):
if model._meta.app_label == 'app2':
return 'mydb2'
return None
def db_for_write(self,model, **hints):
if model._meta.app_label == 'app2':
return 'mydb2'
return None
def allow_relation(self,obj1, obj2, **hints):
if obj1._meta.app_label == 'app2' and \
obj2._meta.app_label == 'app2':
return True
return None
def allow_syncdb(self,db, model):
if db == 'mydb2':
if model._meta.app_label == 'app2':
return True
elif model._meta.app_label == 'app2':
return False
return None
After this I expect that when I run commands makemigrations and migrate I would get 2 different tables in 2 different schemas?
So:
$ python3 manage.py makemigrations
Migrations for 'app1':
app1/migrations/0001_initial.py
- Create model Model1
Migrations for 'app2':
app2/migrations/0001_initial.py
- Create model Model2
$ python3 manage.py migrate
Operations to perform:
Apply all migrations: admin, app1, app2, auth, contenttypes, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
...
Applying sessions.0001_initial... OK
But there are no tables created except django_migrations in both schemas.
If I use command:
$ python3 manage.py migrate --database=mydb1
then both model tables are created in the APP1DB.
mysql> SHOW TABLES;
+----------------------------+
| Tables_in_APP1DB |
+----------------------------+
| app1_model1 |
| app2_model2 |
| auth_group |
So how to solve this?
According to the documentation database routers may implement the method allow_migrate(db, app_label, model_name=None, **hints) in order to determine whether a certain migration should be performed or not. The methods of custom database routers are always invoked through the default database router django.db.router:
def allow_migrate(self, db, app_label, **hints):
for router in self.routers:
try:
method = router.allow_migrate
except AttributeError:
# If the router doesn't have a method, skip to the next one.
continue
[...]
return True
Because your routers don't define such a method it simply returns True at the end of the method and therefore applies the requested migration to the specified database.
You can achieve the separation of apps across the different databases by defining this method:
class App1DBRouter(object):
[...]
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'app1':
return db == 'mydb1'
return None
And similarly for app2.
This drives me mad... I'm reorganizing an existing Django project using the following structure:
[project_abc]
[app]
[core]
[app1]
admin.py
models.py
...
[app2]
admin.py
models.py
...
... etc ... there's a total of 9 apps
[rest]
... rest api stuff, non-db related ...
[mobile]
... mobile stuff, non-db related ...
[
south
tastypie
[project_abc]
settings.py
urls.py
manage.py
All apps with models that require database access have been added to settings.py:
INSTALLED_APPS = (
'django.contrib.admin',
'[app].[core].[app1]',
'[app].[core].[app2]',
...
'tastypie',
'south'
)
Each model class has a Meta class like:
class Meta:
app_label=[app] # this points to the top level above [core]
Directories [app], [core] and subsequent [app] directories have an __init__.py file in them.
When I run syncdb, it happily ignores my apps under [core] however the tables for other apps like tastypie and south get created properly.
When I run manage.py validate it returns 0 errors found
I've read probably all posts and hints on topics related to syncdb but unfortunately to no avail. I'm obviously missing something, but cannot figure out what it is....
I can't fully understand which [app] is set in models Meta, but note that django syncdb uses django.db.models.get_apps to find projects' applications. Latter interspects apps from INSTALLED_APPS, and explicetely tries to load apps' models module with
models = import_module('.models', app_name)
So applications outside INSTALLED_APPS won't have tables synced.
Second, django loads all the models with django.db.models.get_apps for each found application, and latter turn introspects AppCache.apps_models (that cache is, as far as I remember, populated with register_models by model constructor). So all the imported models corresponding to valid applications are processed.
I guess you have to ensure that [app] from models._Meta:
contains models.py (possibly empty) which will make it a django application;
is mentioned in INSTALLED_APPS, to be asseccible with get_apps function.
I have a reusable application. In this app, some models need to be localized and I am using the django-modeltranslation app for it.
Using django-modeltranslation cause the south migrations to include the localized field in the model defintion.
For example, I i have the following model:
class MyModel(models.Model):
name = models.CharField(...)
And the following translation.py file
class MyModelOptions(TranslationOptions):
fields = ('name',)
translator.register(MyModel, MyModelOptions)
and two languages, fr and en, defined in my settings.py
If I run a south schemamigration on this app, south will add the name_fr and name_en field to the model definition of the migration
class Migration(SchemaMigration):
def forwards(self, orm):
#Here the columns are created depending but It can be managed for all languages in settings.LANGUAGES
for (lang, _x) in settings.LANGUAGES:
#create the column for the language
def backwards(self, orm):
#Simimar workaround than forwards can be implemented
models = {
'myapp.mymodel': {
'Meta': {'object_name': 'MyModel'},
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
#The following items are the model definition and can not be generated from settings.LANGUAGES
'name_en': ('django.db.models.fields.CharField', [], {'max_length': '100', 'null': 'True', 'blank': 'True'}),
'name_fr': ('django.db.models.fields.CharField', [], {'max_length': '100', 'null': 'True', 'blank': 'True'}),
}
As far as I know, this model definition is generated by south in a hard coded way.
As a consequence, it is difficult to maintain the south migrations for a reusable app using django-modeltranslation because there is no way to know in advance what are the languages defined in the settings.py of the project.
What would you recommend to manage this issue?
To be honest I wouldn't add those migrations to the package at all, one shouldn't force fixtures upon a 3rd party user. A nice way could be to create a demo project within the package and add proper documentation, e.g. (django 1.4+)
repository_root/
example/
example/
__init__.py
urls.py
settings.py
static/
js/
fixtures/
data.json
migrations/
reusable_app/
__init__.py
0001_initial.py
manage.py
reusable_app/
models.py
urls.py
views.py
admin.py
Add a few settings to settings.py to keep things clean
def rel(*x):
return os.path.join(os.path.abspath(os.path.dirname(__file__)), *x)
FIXTURE_DIRS = (
rel('fixtures'),
)
SOUTH_MIGRATION_MODULES = {
'reusable_app': 'example.migrations.reusable_app',
}
Make sure you add the following line after your import statements in manage.py to make sure you're working on the local reusable_app not the (installed) one in site-packages
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
The option I choosed is to disable (remove from INSTALLED_APPS ) models translation when making a schemamigration and let sync_translation_fields manage the creation of missing translation fields.
It seems an acceptable method : https://github.com/deschler/django-modeltranslation/issues/106#issuecomment-33875679