I have a simple integration test in Django that spawns a single Celery worker to run a job, which writes a record to the database. The Django thread also writes a record to the database. Because it's a test, I use the default in-memory sqlite3 database. There are no transactions being used.
I often get this error:
django.db.utils.OperationalError: database table is locked
which according to the Django docs is due to one connection timing out while waiting for another to finish. It's "more concurrency than sqlite can handle in default configuration". This seems strange given that it's two records in two threads. Nevertheless, the same docs say to increase the timeout option to force connections to wait longer. Ok, I change my database settings to this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
'OPTIONS': {'timeout': 10000000},
}
}
This has no effect. The error still appears and it clearly has not waited 1e7 seconds or 1e7 milliseconds or 1e7 microseconds before doing so. Is there an additional setting I'm missing?
I have tried both Python 3.5 and Python 3.6 and both Django 1.11 and Django 2.0.
I had the same issue and my experiments gave me the following:
I've figured out that Django uses in-memory SQLite DB in the test mode until you explicitly change this. That explains why I only see that problem in my unit tests. To force Django to use SQLite DB in the file set DATABASES->TEST->NAME explicitly in your settings.py. For example like this:
DATABASES = {
'default': {
...
'TEST': {
'NAME': 'testdb.sqlite3',
},
},
}
Setting timeout value larger than 2147483.647 (looks familiar, right? :-) ) disables timeout (or sets it to negligibly small value).
As far as I understand, the root of the problem is that when SQLite uses the shared cache the timeout value is not respected at all.
Related
For performance tuning reasons, I want to run Django tests against a copy of my production database. As I understand it, this should be possible by:
(1) adjusting Django settings like
DATABASES = {
'default': {
...
'TEST': {
'NAME': 'my_database_copy',
},
}
}
and (2) using the --keepdb flag, as in python manage.py test --keepdb.[1]
But when I do this, the process hangs, looking like this:
bash-4.2$ python manage.py test --keepdb
Using existing test database for alias 'default'...
(The process won't close with ctrl+c. I'm using Docker, and I stop it by restarting Docker.)
There are no unapplied migrations for the database, and the test command (python manage.py test) works fine, if --keepdb is omitted.
I confirmed that the database copy is properly restored and accessible because I can access it when I run python manage.py shell.
[1] https://docs.djangoproject.com/en/3.1/topics/testing/overview/#preserving-the-test-database
Adjust the settings dictionary by adding the SERIALIZE key, like this:
DATABASES = {
'default': {
...
'TEST': {
'NAME': 'my_database_copy',
'SERIALIZE': False,
},
}
}
When SERIALIZE is True (the default), Django tries to read a copy of the whole database into memory as a string. See[1]. This considered helpful for tests when the database engine does not support transactions, but in my case crashed due to insufficient memory. Deactivating this behavior through settings is covered here[2].
[1] https://github.com/django/django/blob/d5b526bf78a9e5d9760e0c0f7647622bf47782fe/django/db/backends/base/creation.py#L73
[2] https://docs.djangoproject.com/en/3.1/ref/settings/#serialize
I have a project running in Django and connecting do SQLAlchemy ORM via sessionmaker, as showed below. It basicly handles http methods with a specified API (GET, POST, DELETE) and returns, posts, updates or deletes db entries.
from sqlalchemy import create_engine
from sqlalchemy.orm.session import sessionmaker
self.session = sessionmaker(
bind=create_engine('mysql+pymysql://user:pw#127.0.0.1/db')
Under myproject.settings I am using defaults like 'ENGINE': 'django.db.backends.sqlite3',.
I would like to test whether the API is working as intended by simply interating through all possible methods and URIs which seem necessary to test. Testing is done with Django's TestCase class and its Client module. Works quite fine.
My Problem:
It is altering (especially deleting and updating columns within) the real db. Rather than using the "created and destroyed test_db" as Django's test output might indicate it is using the real db.
I kinda get why (I am bypassing Django's built-in db-connection with my SQLAlchemy connection), but I am interested in how to fix this, i.e. using a true test_db.
Currently I am using a read-only mysql-user for testing, but that prevents me from testing actual POST and DELETE requests.
I could try to use a different db for testing by mocking, but I would prefer another solution (I would have to create a dummy db from the real one, every time I ran a test)
PS: If you feel I have not provided enough code, give me a hint. But I feel people might get the idea of my problem and the solution is probably the mysql integration into Django, which I had not the need to do properly yet. Or more accuratly which I could not get working every time I tried.
EDIT: When trying to configure my database to
DATABASES = {
'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
'ENGINE': 'django.db.backends.mysql',
'NAME': 'db',
'USER': 'user',
'PASSWORD': 'pw',
'HOST': '127.0.0.1',
}
}
I get django.db.utils.OperationalError: (2006, 'SSL connection error: SSL_CTX_set_tmp_dh failed') which I is due to not using pymysql here, I figure.
I am building a Django project that uses a relational DB (for development purposes SQLite) and an non-relational DB (OrientDB). This is my first time a using non-relational DB and I m having difficulty getting it set up with Django.
The use of OrientDB in my project is solely to keep track of friend relationships and friend-of-friend relationships, while all other user data is being stored in my relational DB.
I know I need to register the DB in my settings file. I am trying to do something like this:
#setting.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
},
'friends': {
'NAME': 'friends',
'ENGINE': 'django.db.backends.orientdb',
'USER': 'root',
'PASSWORD': 'hello',
'HOST': '',
'PORT': '2480',
}
}
When I do this, however, I get the error:
No module named 'django.db.backends.orientdb'
Is this backend module something I have to create myself or can I manually connect to the DB in my code whenever I need something specific done? For example, whenever someone creates a new user in my SQLite DB, can I use a Signal post_save to
connect to OrientDb,
create a friend instance in Orient DB, and
disconnects from OrientDB?
It seems like there ought to be a much cleaner way of doing this.
This is almost certainly something you'll need to build yourself, though your use case doesn't sound like it requires a whole Django backend. A few manual queries might be enough.
Django officially supports PostgreSQL, MySQL, SQLite, and Oracle. There are third-party backends for SAP SQL Anywhere, IBM DB2, Microsoft SQL Server, Firebird, and ODBC.
There is an abandoned project that attempted to provide an OrientDB backend for Django, but it hasn't been updated in quite a long time and likely needs a lot of love:
This project isn't maintained anymore, feel free to fork and keep it alive.
No matter how you choose to proceed you should probably take a look at OrientDB's Python library.
I'm trying to implement a failover strategy when my MySQL backend is down in Celery.
I found in this other stack overflow answer that failover is made possible in SQLAlchemy. However, I couldn't write the same behavior in Celery using sqlalchmey_engine_options
__app.conf.result_backend = 'db+mysql://scott:tiger#localhost/foo'
__app.conf.sqlalchmey_engine_options = {
'connect_args': {
'failover': [{
'user': 'root',
'password': 'password',
'host': 'http://other_db.com',
'database': 'dbname'
}]
}
}
What I'm trying to do is if the first backend scott:tiger does not respond, then it switches to root:password backend.
There is definitely more than one way to achieve failover. You could start with simple try..except and handle situation when your prefered backend is not responding, in simplest (and probably not very pythonic) way you could try something like this:
try:
# initialise your SQL here and also set the connection up
except:
# initialise your backup SQL here
You could also move your backend selection to the infrastructure so it's transparent from your application perspective, i.e. by using session pooling system (I am not MySQL user but in PostgreSQL world we have pgpool).
--- edit ---
I realised you probably want to have your database session and connection handled by celery itself. So very likely above does not answer your question directly, in my simple project I initialise database connection within tasks that require it as in my particular case most tasks do not require database at all.
I could like get my celery worker process to talk to the django test database.
Its an oracle database, so I believe the database/user is already created.
I am just trying to figure out what to pass the Celery/App configuration to get it to talk to the "TEST" database.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.oracle',
.............
'TEST': {
'USER': WIT_TEST_DB_USER,
'PASSWORD': WIT_TEST_DB_USER,
}
}
}
I have seen a stackoverflow article that talks about passing the settings.conf from the parent test setup() to the worker process. That may be necessary when the test database file is automatically generated in case of sqllite databases.
In my case, its a well defined oracle test database that I think is already part of the config/settings files.
so I am looking for a way to directly start the work process independent of the testrunner/testcase code.
Can some one suggest an approach to doing this?
You are regarding your test database as an ordinary database. So I think the best solution would be to define your test database as the default database under DATABASES settings in a separate settings file. And when running your worker you can pass the specific new settings file to your worker like this:
export DJANGO_SETTINGS_MODULE='[python path to your celery specific settings file]'
# the command to run your celery worker