I have made some repetitive operations in my application (testing it), and suddenly I’m getting a weird error:
OperationalError: database is locked
I've restarted the server, but the error persists. What can it be all about?
From django doc:
SQLite is meant to be a lightweight
database, and thus can't support a
high level of concurrency.
OperationalError: database is locked
errors indicate that your application
is experiencing more concurrency than
sqlite can handle in default
configuration. This error means that
one thread or process has an exclusive
lock on the database connection and
another thread timed out waiting for
the lock the be released.
Python's SQLite wrapper has a default
timeout value that determines how long
the second thread is allowed to wait
on the lock before it times out and
raises the OperationalError: database
is locked error.
If you're getting this error, you can
solve it by:
Switching to another database backend. At a certain point SQLite becomes too "lite" for real-world applications, and these sorts of concurrency errors indicate you've reached that point.
Rewriting your code to reduce concurrency and ensure that database transactions are short-lived.
Increase the default timeout value by setting the timeout database option
http://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errorsoption
In my case, It was because I open the database from SQLite Browser. When I close it from the browser, the problem is gone.
I slightly disagree with the accepted answer which, by quoting this doc, implicitly links OP's problem (Database is locked) to this:
Switching to another database backend. At a certain point SQLite becomes too "lite" for real-world applications, and these sorts of concurrency errors indicate you've reached that point.
This is a bit "too easy" to incriminate SQlite for this problem (which is very powerful when correctly used; it's not only a toy for small databases, fun fact: An SQLite database is limited in size to 140 terabytes ).
Unless you have a very busy server with thousands of connections at the same second, the reason for this Database is locked error is probably more a bad use of the API, than a problem inherent to SQlite which would be "too light". Here are more informations about Implementation Limits for SQLite.
Now the solution:
I had the same problem when I was using two scripts using the same database at the same time:
one was accessing the DB with write operations
the other was accessing the DB in read-only
Solution: always do cursor.close() as soon as possible after having done a (even read-only) query.
Here are more details.
The practical reason for this is often that the python or django shells have opened a request to the DB and it wasn't closed properly; killing your terminal access often frees it up. I had this error on running command line tests today.
Edit: I get periodic upvotes on this. If you'd like to kill access without rebooting the terminal, then from commandline you can do:
from django import db
db.connections.close_all()
As others have told, there is another process that is using the SQLite file and has not closed the connection. In case you are using Linux, you can see which processes are using the file (for example db.sqlite3) using the fuser command as follows:
$ sudo fuser -v db.sqlite3
USER PID ACCESS COMMAND
/path/to/db.sqlite3:
user 955 F.... apache2
If you want to stop the processes to release the lock, use fuser -k which sends the KILL signal to all processes accessing the file:
sudo fuser -k db.sqlite3
Note that this is dangerous as it might stop the web server process in a production server.
Thanks to #cz-game for pointing out fuser!
I got this error when using a database file saved under WSL (\\wsl$ ...) and running a windows python interpreter.
You can either not save the database in your WSL-tree or use a linux based interpreter in your distro.
I encountered this error message in a situation that is not (clearly) addressed by the help info linked in patrick's answer.
When I used transaction.atomic() to wrap a call to FooModel.objects.get_or_create() and called that code simultaneously from two different threads, only one thread would succeed, while the other would get the "database is locked" error. Changing the timeout database option had no effect on the behavior.
I think this is due to the fact that sqlite cannot handle multiple simultaneous writers, so the application must serialize writes on their own.
I solved the problem by using a threading.RLock object instead of transaction.atomic() when my Django app is running with a sqlite backend. That's not entirely equivalent, so you may need to do something else in your application.
Here's my code that runs FooModel.objects.get_or_create simultaneously from two different threads, in case it is helpful:
from concurrent.futures import ThreadPoolExecutor
import configurations
configurations.setup()
from django.db import transaction
from submissions.models import ExerciseCollectionSubmission
def makeSubmission(user_id):
try:
with transaction.atomic():
e, _ = ExerciseCollectionSubmission.objects.get_or_create(
student_id=user_id, exercise_collection_id=172)
except Exception as e:
return f'failed: {e}'
e.delete()
return 'success'
futures = []
with ThreadPoolExecutor(max_workers=2) as executor:
futures.append(executor.submit(makeSubmission, 296))
futures.append(executor.submit(makeSubmission, 297))
for future in futures:
print(future.result())
I was facing this issue in my flask app because I opened the database in SQLite Browser and forgot to write the changes.
If you have also made any changes in SQLite Browser, then click on write changes and everything will be fine.
This also could happen if you are connected to your sqlite db via dbbrowser plugin through pycharm. Disconnection will solve the problem
For me it gets resolved once I closed the django shell which was opened using python manage.py shell
I've got the same error! One of the reasons was the DB connection was not closed.
Therefore, check for unclosed DB connections. Also, check if you have committed the DB before closing the connection.
I had a similar error, right after the first instantiation of Django (v3.0.3). All recommendations here did not work apart from:
deleted the db.sqlite3 file and lose the data there, if any,
python manage.py makemigrations
python manage.py migrate
Btw, if you want to just test PostgreSQL:
docker run --rm --name django-postgres \
-e POSTGRES_PASSWORD=mypassword \
-e PGPORT=5432 \
-e POSTGRES_DB=myproject \
-p 5432:5432 \
postgres:9.6.17-alpine
Change the settings.py to add this DATABASES:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'myproject',
'USER': 'postgres',
'PASSWORD': 'mypassword',
'HOST': 'localhost',
'PORT': '5432',
}
}
...and add database adapter:
pip install psycopg2-binary
Then the usual:
python manage.py makemigrations
python manage.py migrate
Check if your database is opened on another DB Browser.
If it is opened on an other application, then close the application and run the program again.
Just close (stop) and open (start) the database. This solved my problem.
I found this worked for my needs. (thread locking) YMMV
conn = sqlite3.connect(database, timeout=10)
https://docs.python.org/3/library/sqlite3.html
sqlite3.connect(database[, timeout, detect_types, isolation_level, check_same_thread, factory, cached_statements, uri])
When a database is accessed by multiple connections, and one of the processes modifies the database, the SQLite database is locked until that transaction is committed. The timeout parameter specifies how long the connection should wait for the lock to go away until raising an exception. The default for the timeout parameter is 5.0 (five seconds).
In my case, I added a new record manually saved and again through shell tried to add new record this time it works perfectly check it out.
In [7]: from main.models import Flight
In [8]: f = Flight(origin="Florida", destination="Alaska", duration=10)
In [9]: f.save()
In [10]: Flight.objects.all()
Out[10]: <QuerySet [<Flight: Flight object (1)>, <Flight: Flight object (2)>, <Flight: Flight object (3)>, <Flight: Flight object (4)>]>
In my case, I had not saved a database operation I performed within the SQLite Browser. Saving it solved the issue.
A very unusual scenario, which happened to me.
There was infinite recursion, which kept creating the objects.
More specifically, using DRF, I was overriding create method in a view, and I did
def create(self, request, *args, **kwargs):
....
....
return self.create(request, *args, **kwargs)
Already lot of Answers are available here, even I want to share my case , this may help someone..
I have opened the connection in Python API to update values, I'll close connection only after receiving server response. Here what I did was I have opened connection to do some other operation in server as well before closing the connection in Python API.
If you get this error while using manage.py shell, one possible reason is that you have a development server running (manage.py runserver) which is locking the database. Stoping the server while using the shell has always fixed the problem for me.
actually I have faced same problem , when I use "transaction.atomic() with select_for_update() " i got error message "the OperationalError: database is locked" ,
and after many tries / searching / read django docs ,
i found the problem from SQLite itself it is not support select_for_update method as django DOCs says , kindly have a look at the following url and read it deeply:
https://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errors
, and when i moved to MySQL everything goes fine .
as django DOCs also says "database is locked" may happen when database timeout occur ,
they recommend you to change database timeout by setting up the following option :
'OPTIONS': {
# ...
'timeout': 20,
# ...
}
finally, I recommend you to use MySQL/PostgreSQL even if you working on development environment .
I hope this helpful for you .
I got this error when attempting to create a new table in SQLite but the session object contained uncommitted (though flushed) changes.
Make sure to either:
Commit the session(s) before creating a new table
Close all sessions and perform the table creation in a new connection
...
#Shilp Thapak's answer is correct: the reason for the error is that you did not write your manual changes to the data in your DB Browser for SQLite before running your application.
If you didn't write the changes in whatever SQL client you are using, you can still create the engine but
engine.connect()
will throw the operational error about the database being locked.
You can check whether your engine can connect by checking the existence of a rollback journal. The default mode of a rollback journal is to be created and deleted at the start and end of a transaction.
It is exists in the same directory where your database is, it has the same name as the database file and the suffix "-journal" appended.
If the mode is not changed, at Journal mode in Edit pragmas panel in DB Browser for SQLite.
You can check the existence of the temp file like so:
if os.path.isfile('your-database.sqlite-journal'):
print("The database is locked. Please write your changes in your SQL client before proceeding.\n")
Read more about temporary files here.
So no need to close the server or DB Browser for SQLite for that sake. In fact, as long as all the changes are written, you can have several clients connected to the database simultaneously and still run your application at the same time.
For me it was simply because I was accessing the database in SQLite app at the same time of running my Python code to create a new table.
Closing SQLite until the code is done solved my issue.
UPDATE django version 2.1.7
I got this error sqlite3.OperationalError: database is locked using pytest with django.
Solution:
If we are using #pytest.mark.django_db decorator. What it does is create a in-memory-db for testing.
Named: file:memorydb_default?mode=memory&cache=shared We can get this name with:
from django.db import connection
db_path = connection.settings_dict['NAME']
To access this database and also edit it, do:
Connect to the data base:
with sqlite3.connect(db_path, uri=True) as conn:
c = conn.cursor()
Use uri=True to specifies the disk file that is the SQLite database to be opened.
To avoid the error activate transactions in the decorator:
#pytest.mark.django_db(transaction=True)
Final function:
from django.db import connection
#pytest.mark.django_db(transaction=True)
def test_mytest():
db_path = connection.settings_dict['NAME']
with sqlite3.connect(db_path, uri=True) as conn:
c = conn.cursor()
c.execute('my amazing query')
conn.commit()
assert ... == ....
Just reboot your server, it will clear all current processes that have your database locked.
I just needed to add alias sqlite='sqlite3' to my ~/.zshrc
I then deleted the partially-failed creation of the virtualenv in ~/.pyenv/versions/new-virtualenv and reran pyenv virtualenv <name> and it worked swimmingly
try this command:
sudo fuser -k 8000/tcp
I'm actually a php(CodeIgniter) web developer though I love python I just installed Bitnami's Django Stack which has Apache, MySQL, PostgreSQL and Python 2.7.9 with Django installed. During installation itself it generated a simple Django project.
Though it looked familiar to me I started adding some lines of codes to it but when I save it and refresh the page or even restart the browser I found that python instance is still running the old script. The script updates only when I restart start the Apache Server(I believe that's where the Python instance got terminated).
So, to clarify this problem with Python I created a simple view and URLed it to r'^test/'
from django.http import HttpResponse
i = 0
def test_view(request):
global i
i += 1
return HttpResponse(str(i))
Then I found that even switching between different browser the i value keep on increasing i.e increasing value continues with the other browse.
So, can anyone tell me is this a default behavior of Django or is there something wrong with my Apache installation.
This is the default behavior, it may reset if you were running with gunicorn and killing workers after X requests or so, I don't remember. It's like this because the app continues to run after a request has been served.
Its been a while I've worked with PHP but I believe, a request comes in, php starts running a script which returns output and then that script terminates. Special global variables like $_SESSION aside, nothing can really cross requests.
Your Django app starts up and continues to run unless something tells it to reload (when running with ./manage.py runserver it will reload whenever it detects changes to the code, this is what you want during development).
If you are interested in per visitor data see session data. It would look something like:
request.session['i'] = request.session.get('i', 0) + 1
You can store data in there for the visitor and it will stick around until they lose their session.
I'm using the Django ORM in a non-Django application and would like to turn on the DEBUG setting so that I can periodically log my queries. So I have something vaguely like this:
from django.db import connection
def thread_main_loop():
while keep_going:
connection.queries[:] = []
do_something()
some_logging_function(connection.queries)
I would like to do this on my production server, but the doc warns, "It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you are debugging, but on a production server, it will rapidly consume memory."
Because the connection.queries list is cleared every time through the main loop of each thread, I believe that Django query logging will not cause my application to consume memory. Is this correct? And are there any other reasons not to turn DEBUG on in a production environment if I'm only using the Django ORM?
In DEBUG mode any error in your application will lead to the detailed Django stacktrace. This is very undesirable in a production environment as it will probably leak sensitive information that attackers can use against your site. Even if your application seems pretty stable, I wouldn't risk it.
I would rather employ a middleware that somehow logs queries to a file. Or take statistics of the database directly, e.g (for MySQL).
watch -n 1 mysqladmin --user=<user> --password=<password> processlist
Edit:
If you are only using the Django ORM, then afaik only two things will be different:
Queries will be saved with the CursorDebugWrapper
If a query results in a database warning, this will raise an exception.
I have a multit-threaded xmlrpc service running which stores a huge amount of data ~2G in memory. Currently, if I want to update a method the server exposes I have to restart the service. The problem here is that if I restart the service it needs to load all of the data it had in memory back into memory by using a database or using shelved data.
I am using methods like this:
xmlrpc_getUser(self, uid):
return self.users[uid]
What I was hoping I could do is just use these methods as a proxy to another module, so my methods would look more like this
xmlrpc_getUser(self, uid):
return self.proxy.getUser(uid)
This way I could update code on the development server then simply copy my update proxy module to the production server without the need for a restart.
I tried adding
import service_proxy
to the constructor of my xmlrpc service controller, but I think the module is cached and won't reload.
Is there a good way to do this? Thanks.
You could use the reload method. You would need to write some code to check the last modified time of the modules file.
If reload doesn't work, you could try twisted.python.rebuild; your application need not be written in Twisted to use this twisted.python utility.
I also recently saw this livecoding thing ("a code reloading library for Python"), but it talks about a custom module system and I don't know what's going on there.