Develop Reference

Django ORM using external database to unmanaged model in view says relation does not exist

I have two Postgres database connections and when using another than default ORM calls fail on views, but not raw. I am using docker and containers are connected and using Django's own runserver command. Using ORM in django command or django shell works fine, but not in a view.
As a side note, both databases are actually Django-projects, but main project is reading some data directly from another project's database using own unmanaged model.
Python: 3.9.7
Django: 3.2.10
Postgres: 13.3 (main project), 12.2 (side project)
# settings
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'USER': 'pguser',
'PASSWORD': 'pguser',
'NAME': 'mainproject',
'HOST': 'project-db', # docker container
'PORT': '5432',
},
'external': {
'ENGINE': 'django.db.backends.postgresql',
'USER': 'pguser',
'PASSWORD': 'pguser',
'NAME': 'sideproject',
'HOST': 'side-db', # docker container, attached to same network
'PORT': '5432',
},
}
# My unmanaged model
class MyTestModel(models.Model):
​
class Meta:
# table does not exist in 'default', but it does exist in 'external'
db_table = 'my_data_table'
managed = False
# my_data_table is very simple it has id field as integer
# and primary key (works out of the box with django)
# it has normal fields only like IntegerField or CharField
# But even if this model is empty then it returns normally the PK field
# My custom command in Main project
# python manage.py mycommand
# ...
def handle(self, *args, **options):
# This works fine and data from external is populated in MyTestModel
data = MyTestModel.objects.using('external').all()
for x in data:
print(x, vars(x))
# My simple view in Main project
class MyView(TemplateView):
# But using example in views.py, random view class:
def get(self, request, *args, **kwargs):
# with raw works
data = MyTestModel.objects.using('external').raw('select * from my_data_table')
for x in data:
print(x, vars(x))
# This does not work
# throws ProgrammingError: relation "my_data_table" does not exist
data = MyTestModel.objects.using('external').all()
for x in data:
print(x, vars(x))
return super().get(request, *args, **kwargs)
So somehow runserver and views does not generate query correctly when using ORM. It cannot be a connection error, because when running command or view with ".raw()" works.
Now the funny thing is that if I change "db_table" to something what is common in both database lets say "django_content_type" ORM "filter()" and "all()" works in view too. And yes then it actually returns data from correct database. So if main project has 50 content types and side project (external) has 100 content types it actually returns those 100 content types from external.
I have tried everything, rebuilding docker, creating new tables to database directly, force reload postgres user and made sure all is owner permissions and all permissions (should be ok anyway because command side works). I even tried to use another database.

I know I didn't post the full settings and my local settings which could have helped more to solve this case.
But I noticed that I had installed locally Django Silk, which captures the request and tries to analyze called database queries. Looks like it may have been loaded too early or it doesn't like external databases. But disabling Django silk from installed apps and removing it's middleware removed the problem.

Specifying Readonly access for Django.db connection object

I have a series of integration-level tests that are being run as a management command in my Django project. These tests are verifying the integrity of a large amount of weather data ingested from external sources into my database. Because I have such a large amount of data, I really have to test against my production database for the tests to be meaningful. What I'm trying to figure out is how I can define a read-only database connection that is specific to that command or connection object. I should also add that these tests can't go through the ORM, so I need to execute raw SQL.
The structure of my test looks like this
class Command(BaseCommand):
help = 'Runs Integration Tests and Query Tests against Prod Database'
def handle(self,*args, **options):
suite = unittest.TestLoader().loadTestsFromTestCase(TestWeatherModel)
ret = unittest.TextTestRunner().run(suite)
if(len(ret.failures) != 0):
sys.exit(1)
else:
sys.exit(0)
class TestWeatherModel(unittest.TestCase):
def testCollectWeatherDataHist(self):
wm = WeatherManager()
wm.CollectWeatherData()
self.assertTrue(wm.weatherData is not None)
And the WeatherManager.CollectWeatherData() method would look like this:
def CollecWeatherData(self):
cur = connection.cursor()
cur.execute(<Raw SQL Query>)
wm.WeatherData = cur.fetchall()
cur.close()
I want to somehow idiot-proof this, so that someone else (or me) can't come along later and accidentally write a test that would modify the production database.

You can achieve this by hooking into Django's connection_created signal, and
then making the transaction read-only.
The following works for PostgreSQL:
from django.db.backends.signals import connection_created
class MyappConfig(AppConfig):
def ready(self):
def connection_created_handler(connection, **kwargs):
with connection.cursor() as cursor:
cursor.execute('SET default_transaction_read_only = true;')
connection_created.connect(connection_created_handler, weak=False)
This can be useful for some specific Django settings (e.g. to run development
code with runserver against the production DB), where you do not want to
create a real read-only DB user.

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'mydb',
'USER': 'myusername',
'PASSWORD': 'mypassword',
'HOST': 'myhost',
'OPTIONS': {
'options': '-c default_transaction_read_only=on'
}
}
}
Source: https://nejc.saje.info/django-postgresql-readonly.html

Man, once again, I should read the docs more carefully before I post questions here. I can define a readonly connection to my production database in the settings file, and then straight from the docs:
If you are using more than one database, you can use django.db.connections to obtain the connection (and cursor) for a specific database. django.db.connections is a dictionary-like object that allows you to retrieve a specific connection using its alias:
from django.db import connections
cursor = connections['my_db_alias'].cursor()
# Your code here...

If you add a serializer for you model, you could specialized in the serializer that is working in readonly mode
class AccountSerializer(serializers.ModelSerializer):
class Meta:
model = Account
fields = ('id', 'account_name', 'users', 'created')
read_only_fields = ('account_name',)
from http://www.django-rest-framework.org/api-guide/serializers/#specifying-read-only-fields

Is there any method in django to use database without creating models.py file?

I am having a database with some data filled in it and i want to use it in my new django app. So is their any way to use data of database in my django app.Actually i don't want to make any changes in my old database and only want to use its data. Anybody please suggest me what will be the better approach to do this.
While serching i also found a command-inspectdb
which can generate model.py file from database, but their are some issues with it that it does'nt map the foreign key in model.py, we need to rearrange our classes in model.py file and some more. So i am searching for some other alternative.

You could access data from legacy database using connection.cursor() from django.db module.
If you have two dabases
DATABASES = {
'default': {
'NAME': 'new_database',
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'USER': '',
'PASSWORD': ''
},
'old': {
'NAME': 'old_database',
'ENGINE': 'django.db.backends.mysql',
'USER': '',
'PASSWORD': ''
}
}
...
from django.db import connections
cursor = connections['old'].cursor()
cursor.execute("SELECT...")
cursor.fetchall()
refer to docs:
Executing custom SQL directly
Multiple databases
But if you want to modify data in your old database it is better idea to create models.py file and use it as always. Using inspectdb or not is up to you. For example you cold generate model using inpsectdb in separate temporary project, make dumpdata to create json files and upload data to your active project somehow.

Force strict sql mode in django

I'm trying to force my django project to always use strict sql_mode. Is there another way than putting the following in manage.py? It seems overly complicated.
def set_strict_sql_mode(sender, **kwargs):
from django.conf import settings
if settings.DATABASES['default']['ENGINE'] == 'django.db.backends.mysql':
from django.db import connection
cursor = connection.cursor()
cursor.execute('SET session sql_mode=traditional')
from django.core.signals import request_started
request_started.connect(set_strict_sql_mode)

Actually asking proved to be a good rubber duck. Just after asking, I found the custom database OPTIONS one can supply in the DATABASES settings like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'sql_mode': 'traditional',
}
}
}
Hope it helps anyone!

You can also try with Adding below option in Database []
'OPTIONS': {
'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
},
Its working.

https://docs.djangoproject.com/en/1.11/ref/databases/#mysql-sql-mode
Setting sql_mode
From MySQL 5.7 onwards and on fresh installs of MySQL 5.6, the default
value of the sql_mode option contains STRICT_TRANS_TABLES. That option
escalates warnings into errors when data are truncated upon insertion,
so Django highly recommends activating a strict mode for MySQL to
prevent data loss (either STRICT_TRANS_TABLES or STRICT_ALL_TABLES).
If you need to customize the SQL mode, you can set the sql_mode
variable like other MySQL options: either in a config file or with the
entry 'init_command': "SET sql_mode='STRICT_TRANS_TABLES'" in the
OPTIONS part of your database configuration in DATABASES.

If you need to union the queryset, you can use python chain inster than union.
from itertools import chain
gobj_list = user.groups.all()
robj_list = [obj.roles.all() for obj in gobj_list]
ret_list = chain(*robj_list)

How to get the database where a model instance was saved to in Django?

I have a django application using multiple databases. Given an instance of a model, how can I obtain the database where it is stored (if any)? I need this to save another object to the same database as the first.
def add_ducks_to_hunt(hunter):
db = # the hunter's db
duck = Duck()
duck.save(using=db)

Use _state.db, as in:
my_obj = MyModel.objects.get(pk=1)
my_obj._state.db
This is shown in the routing example.

I believe you are looking for settings.py in the project (top level) Django directory. If not, would you post more information?
Here is a bit of my settings.py file in the Django project directory /home/amr/django/amr.
import os
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
# ('Somone R. Somebody'so-forath-and-so-and-so#somewhere.com'),
)
MANAGERS = ADMINS
DATABASES = {
'default': {
#'ENGINE': 'django.db.backends.', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'ENGINE': 'mysql',
'NAME': 'some-server-name', # Or path to database file if using sqlite3.
'USER': '<user>', # Not used with sqlite3.
'PASSWORD': 'xxxxxxxx', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
}
SESSION_COOKIE_AGE = 15 * 60 * 60 # Age of cookie, in seconds

If you always save a model in the same db, you could store that info in model class and then use it when needed. Probably not very pythonic, but it may do the trick.
Class Hunter(models.Model):
# ... other stuff
db = 'hunter_db'
Access it then with __class__:
def add_ducks_to_hunt(hunter):
db = hunter.__class__.db
duck = Duck()
duck.save(using=db)
On the other hand, if you save objects of one class in different databases, than I see no other way than to write each object's id and db name at object's save() to a table somewhere you can always access it and use that info to load the object.

There is no way to directly get the info from a model instance itself. Ah, _state, yes. its You should explicitly pass in 'using' parameter, just like those built-in methods. Or design router which could deduce the DB to use by check some facts of the model instance, here the hunter.

Assuming you have multiple databases, and you've defined a router, then you can just query your router like:
from myrouters import MyRouter
from myapp.models import Duck
using = MyRouter().db_for_write(Duck)
duck = Duck()
duck.save(using=using)
That said, I'm not sure why you'd need to do this. If the model exists on one database, and you've properly defined a router, then this should be unnecessary. Django will automatically lookup the correct default value for using. If the model exists on multiple databases simultaneously and want to control the specific database where it's written to, then you should already know which database you want to use and shouldn't need to query it.