I have the following code in a Django project (within the create method of a Django Rest Framework serializer)
def create(self, validated_data):
<...>
log.info("\n\n\n")
log.info(f"django model: {self.Meta.model}")
log.info("CREATING CASE NOW .....")
case = self.Meta.model(**kwargs)
log.info(f"Case to be saved: {case}")
case.save()
log.info(f"Case object Created: {case}")
When I'm posting to the endpoint, it's just freezing up completely on .save(). Here's example output:
2020-06-15 02:47:46,008 - serializers - INFO ===> django model: <class 'citator.models.InternalCase'>
2020-06-15 02:47:46,008 - serializers - INFO ===> django model: <class 'citator.models.InternalCase'>
2020-06-15 02:47:46,009 - serializers - INFO ===> CREATING CASE NOW .....
2020-06-15 02:47:46,009 - serializers - INFO ===> CREATING CASE NOW .....
2020-06-15 02:47:46,010 - serializers - INFO ===> Case to be saved: seychelles8698
2020-06-15 02:47:46,010 - serializers - INFO ===> Case to be saved: seychelles8698
No error is thrown and the connection isn't broken. How can I debug this? Is there a way to get logging from the save method?
The error likely unrelated to the use of the Django rest serializers as the code that hangs simple creates a new model and saves it. Now you did not specify how kwargs is defined, but the most likely candidate is that it gets stuck talking to the DB.
To debug the code, you should learn how to step in the code. There are a number of options depending on your preferences.
Visual studio code
Install the debugpy package.
Run python3 -m debugpy --listen localhost:12345 --pid <pid_of_django_process>
Run the "Python: Remote Attach" command.
CLI
Before the line case.save() do
import pdb; pdb.set_trace()
This assumes you are running the Django server interactively and not e.g. through gunicorn. You will get a debug console right before the save line. When the console appears, type 'c' and press enter to continue execution. Then press Ctrl+C when the process appears stuck. Type bt to find out what goes on in the process.
Native code
If the stack trace points to native code, you could switch over to gdb. To debug this (make sure to exit any Python debugger or restart the process without a debugger). Run
gdb -p <pid_of_django>
when the process appears stuck. Then type 'bt' and press enter to get a native traceback of what is going on. This should help you identifiy e.g. database clients acting up.
It is very probable that Django is waiting for a response from database server and it is a configuration problem, not a problem in the Python code where it froze. It is better to check and exclude this possibility before debugging anything. For example it is possible that a table is locked or an updated row is locked by another frozen process and the timeout in the database for waiting for end of lock is long and also the timeout of Django waiting for the database response is very long or infinite.
It is confirmed if a similar save operation takes an abnormally long time in another database client, preferably in your favorite database manager.
Waiting for socket responce is excluded if you see CPU % activity of the locked Python process.
It may be easier explored if you can reproduce the problem in CLI by python manage.py shell or python manage.py runserver --nothreading --nothreading. Then you can press Ctrl+C and maybe after some time Ctrl+C again. If you are lucky you kill the process and will see a KeyboardInterrupt with a traceback. It helps you identify if the process was waiting for something else than for a database server socket response.
Another possible cause in Django could be related to a custom callback code connected to a pre_save or post_save signal.
Instead of plain python manage.py ... you can run python -m pdb manage.py ... and optionally set a break point or simply press "c" "Enter" (continue). The process will run and will be not killed after any exception, but stay in pdb (the native Python DeBugger).
I have made some repetitive operations in my application (testing it), and suddenly I’m getting a weird error:
OperationalError: database is locked
I've restarted the server, but the error persists. What can it be all about?
From django doc:
SQLite is meant to be a lightweight
database, and thus can't support a
high level of concurrency.
OperationalError: database is locked
errors indicate that your application
is experiencing more concurrency than
sqlite can handle in default
configuration. This error means that
one thread or process has an exclusive
lock on the database connection and
another thread timed out waiting for
the lock the be released.
Python's SQLite wrapper has a default
timeout value that determines how long
the second thread is allowed to wait
on the lock before it times out and
raises the OperationalError: database
is locked error.
If you're getting this error, you can
solve it by:
Switching to another database backend. At a certain point SQLite becomes too "lite" for real-world applications, and these sorts of concurrency errors indicate you've reached that point.
Rewriting your code to reduce concurrency and ensure that database transactions are short-lived.
Increase the default timeout value by setting the timeout database option
http://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errorsoption
In my case, It was because I open the database from SQLite Browser. When I close it from the browser, the problem is gone.
I slightly disagree with the accepted answer which, by quoting this doc, implicitly links OP's problem (Database is locked) to this:
Switching to another database backend. At a certain point SQLite becomes too "lite" for real-world applications, and these sorts of concurrency errors indicate you've reached that point.
This is a bit "too easy" to incriminate SQlite for this problem (which is very powerful when correctly used; it's not only a toy for small databases, fun fact: An SQLite database is limited in size to 140 terabytes ).
Unless you have a very busy server with thousands of connections at the same second, the reason for this Database is locked error is probably more a bad use of the API, than a problem inherent to SQlite which would be "too light". Here are more informations about Implementation Limits for SQLite.
Now the solution:
I had the same problem when I was using two scripts using the same database at the same time:
one was accessing the DB with write operations
the other was accessing the DB in read-only
Solution: always do cursor.close() as soon as possible after having done a (even read-only) query.
Here are more details.
The practical reason for this is often that the python or django shells have opened a request to the DB and it wasn't closed properly; killing your terminal access often frees it up. I had this error on running command line tests today.
Edit: I get periodic upvotes on this. If you'd like to kill access without rebooting the terminal, then from commandline you can do:
from django import db
db.connections.close_all()
As others have told, there is another process that is using the SQLite file and has not closed the connection. In case you are using Linux, you can see which processes are using the file (for example db.sqlite3) using the fuser command as follows:
$ sudo fuser -v db.sqlite3
USER PID ACCESS COMMAND
/path/to/db.sqlite3:
user 955 F.... apache2
If you want to stop the processes to release the lock, use fuser -k which sends the KILL signal to all processes accessing the file:
sudo fuser -k db.sqlite3
Note that this is dangerous as it might stop the web server process in a production server.
Thanks to #cz-game for pointing out fuser!
I got this error when using a database file saved under WSL (\\wsl$ ...) and running a windows python interpreter.
You can either not save the database in your WSL-tree or use a linux based interpreter in your distro.
I encountered this error message in a situation that is not (clearly) addressed by the help info linked in patrick's answer.
When I used transaction.atomic() to wrap a call to FooModel.objects.get_or_create() and called that code simultaneously from two different threads, only one thread would succeed, while the other would get the "database is locked" error. Changing the timeout database option had no effect on the behavior.
I think this is due to the fact that sqlite cannot handle multiple simultaneous writers, so the application must serialize writes on their own.
I solved the problem by using a threading.RLock object instead of transaction.atomic() when my Django app is running with a sqlite backend. That's not entirely equivalent, so you may need to do something else in your application.
Here's my code that runs FooModel.objects.get_or_create simultaneously from two different threads, in case it is helpful:
from concurrent.futures import ThreadPoolExecutor
import configurations
configurations.setup()
from django.db import transaction
from submissions.models import ExerciseCollectionSubmission
def makeSubmission(user_id):
try:
with transaction.atomic():
e, _ = ExerciseCollectionSubmission.objects.get_or_create(
student_id=user_id, exercise_collection_id=172)
except Exception as e:
return f'failed: {e}'
e.delete()
return 'success'
futures = []
with ThreadPoolExecutor(max_workers=2) as executor:
futures.append(executor.submit(makeSubmission, 296))
futures.append(executor.submit(makeSubmission, 297))
for future in futures:
print(future.result())
I was facing this issue in my flask app because I opened the database in SQLite Browser and forgot to write the changes.
If you have also made any changes in SQLite Browser, then click on write changes and everything will be fine.
This also could happen if you are connected to your sqlite db via dbbrowser plugin through pycharm. Disconnection will solve the problem
For me it gets resolved once I closed the django shell which was opened using python manage.py shell
I've got the same error! One of the reasons was the DB connection was not closed.
Therefore, check for unclosed DB connections. Also, check if you have committed the DB before closing the connection.
I had a similar error, right after the first instantiation of Django (v3.0.3). All recommendations here did not work apart from:
deleted the db.sqlite3 file and lose the data there, if any,
python manage.py makemigrations
python manage.py migrate
Btw, if you want to just test PostgreSQL:
docker run --rm --name django-postgres \
-e POSTGRES_PASSWORD=mypassword \
-e PGPORT=5432 \
-e POSTGRES_DB=myproject \
-p 5432:5432 \
postgres:9.6.17-alpine
Change the settings.py to add this DATABASES:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'myproject',
'USER': 'postgres',
'PASSWORD': 'mypassword',
'HOST': 'localhost',
'PORT': '5432',
}
}
...and add database adapter:
pip install psycopg2-binary
Then the usual:
python manage.py makemigrations
python manage.py migrate
Check if your database is opened on another DB Browser.
If it is opened on an other application, then close the application and run the program again.
Just close (stop) and open (start) the database. This solved my problem.
I found this worked for my needs. (thread locking) YMMV
conn = sqlite3.connect(database, timeout=10)
https://docs.python.org/3/library/sqlite3.html
sqlite3.connect(database[, timeout, detect_types, isolation_level, check_same_thread, factory, cached_statements, uri])
When a database is accessed by multiple connections, and one of the processes modifies the database, the SQLite database is locked until that transaction is committed. The timeout parameter specifies how long the connection should wait for the lock to go away until raising an exception. The default for the timeout parameter is 5.0 (five seconds).
In my case, I added a new record manually saved and again through shell tried to add new record this time it works perfectly check it out.
In [7]: from main.models import Flight
In [8]: f = Flight(origin="Florida", destination="Alaska", duration=10)
In [9]: f.save()
In [10]: Flight.objects.all()
Out[10]: <QuerySet [<Flight: Flight object (1)>, <Flight: Flight object (2)>, <Flight: Flight object (3)>, <Flight: Flight object (4)>]>
In my case, I had not saved a database operation I performed within the SQLite Browser. Saving it solved the issue.
A very unusual scenario, which happened to me.
There was infinite recursion, which kept creating the objects.
More specifically, using DRF, I was overriding create method in a view, and I did
def create(self, request, *args, **kwargs):
....
....
return self.create(request, *args, **kwargs)
Already lot of Answers are available here, even I want to share my case , this may help someone..
I have opened the connection in Python API to update values, I'll close connection only after receiving server response. Here what I did was I have opened connection to do some other operation in server as well before closing the connection in Python API.
If you get this error while using manage.py shell, one possible reason is that you have a development server running (manage.py runserver) which is locking the database. Stoping the server while using the shell has always fixed the problem for me.
actually I have faced same problem , when I use "transaction.atomic() with select_for_update() " i got error message "the OperationalError: database is locked" ,
and after many tries / searching / read django docs ,
i found the problem from SQLite itself it is not support select_for_update method as django DOCs says , kindly have a look at the following url and read it deeply:
https://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errors
, and when i moved to MySQL everything goes fine .
as django DOCs also says "database is locked" may happen when database timeout occur ,
they recommend you to change database timeout by setting up the following option :
'OPTIONS': {
# ...
'timeout': 20,
# ...
}
finally, I recommend you to use MySQL/PostgreSQL even if you working on development environment .
I hope this helpful for you .
I got this error when attempting to create a new table in SQLite but the session object contained uncommitted (though flushed) changes.
Make sure to either:
Commit the session(s) before creating a new table
Close all sessions and perform the table creation in a new connection
...
#Shilp Thapak's answer is correct: the reason for the error is that you did not write your manual changes to the data in your DB Browser for SQLite before running your application.
If you didn't write the changes in whatever SQL client you are using, you can still create the engine but
engine.connect()
will throw the operational error about the database being locked.
You can check whether your engine can connect by checking the existence of a rollback journal. The default mode of a rollback journal is to be created and deleted at the start and end of a transaction.
It is exists in the same directory where your database is, it has the same name as the database file and the suffix "-journal" appended.
If the mode is not changed, at Journal mode in Edit pragmas panel in DB Browser for SQLite.
You can check the existence of the temp file like so:
if os.path.isfile('your-database.sqlite-journal'):
print("The database is locked. Please write your changes in your SQL client before proceeding.\n")
Read more about temporary files here.
So no need to close the server or DB Browser for SQLite for that sake. In fact, as long as all the changes are written, you can have several clients connected to the database simultaneously and still run your application at the same time.
For me it was simply because I was accessing the database in SQLite app at the same time of running my Python code to create a new table.
Closing SQLite until the code is done solved my issue.
UPDATE django version 2.1.7
I got this error sqlite3.OperationalError: database is locked using pytest with django.
Solution:
If we are using #pytest.mark.django_db decorator. What it does is create a in-memory-db for testing.
Named: file:memorydb_default?mode=memory&cache=shared We can get this name with:
from django.db import connection
db_path = connection.settings_dict['NAME']
To access this database and also edit it, do:
Connect to the data base:
with sqlite3.connect(db_path, uri=True) as conn:
c = conn.cursor()
Use uri=True to specifies the disk file that is the SQLite database to be opened.
To avoid the error activate transactions in the decorator:
#pytest.mark.django_db(transaction=True)
Final function:
from django.db import connection
#pytest.mark.django_db(transaction=True)
def test_mytest():
db_path = connection.settings_dict['NAME']
with sqlite3.connect(db_path, uri=True) as conn:
c = conn.cursor()
c.execute('my amazing query')
conn.commit()
assert ... == ....
Just reboot your server, it will clear all current processes that have your database locked.
I just needed to add alias sqlite='sqlite3' to my ~/.zshrc
I then deleted the partially-failed creation of the virtualenv in ~/.pyenv/versions/new-virtualenv and reran pyenv virtualenv <name> and it worked swimmingly
try this command:
sudo fuser -k 8000/tcp
I am trying to find a bug which happens from time to time on our production server, but could not be reproduced otherwise: some value in the DB gets changed in a way which I don't want it to.
I could write a PostgreSQL trigger which fires if this bug happens, and raise an exception from said trigger. I would see the Python traceback which executes the unwanted SQL statement.
But in this case I don't want to stop the processing of the request.
Is there a way to log the Python/Django traceback from within a PostgreSQL trigger?
I know that this is not trival since the DB code runs under a different linux process with a different user id.
I am using Python, Django, PostgreSQL, Linux.
I guess this is not easy since the DB trigger runs in a different context than the python interpreter.
Please ask if you need further information.
Update
One solution might be to overwrite connection.notices of psycopg2.
Is there a way to log the Python/Django traceback from within a PostgreSQL trigger?
No, there is not
The (SQL) query is executed on the DBMS-server, and so is the code inside the trigger
The Python code is executed on the client which is a different process, possibly executed by a different user, and maybe even on a different machine.
The only connection between the server (which detects the condition) and the client (which needs to perform the stackdump) is the connected socket. You could try to extend the server's reply (if there is one) by some status code, which is used by the client to stackddump itself. This will only work if the trigger is part of the current transaction, not of some unrelated process.
The other way is: massive logging. Make the DBMS write every submitted SQL to its logfile. This can cause huge amounts of log entries, which you have to inspect.
Given this setup
(django/python) -[SQL connection]-> (PostgreSQL server)
your intuition that
I guess this is not easy since the DB trigger runs in a different context than the python interpreter.
is correct. At least, we won't be able to do this exactly the way you want it; not without much acrobatics.
However, there are options, each with drawbacks:
If you are using django with SQLAlchemy, you can register event listeners (either ORM events or Core Events) that detect this bad SQL statement you are hunting, and log a traceback.
Write a wrapper around your SQL driver, check for the bad SQL statement you are hunting, and log the traceback every time it's detected.
Give every SQL transaction, or every django request, an ID (could just be some UUID in werkzeug's request-bound storage manager). From here, we gain more options:
Configure the logger to log this request ID everywhere, and log all SQL statements in SQLAlchemy. This lets you correlate Django requests, and specific function invocations, with SQL statements. You can do this with echo= in SQLAlchemy.
Include this request ID in every SQL statement (extra column?), then log this ID in the PostgreSQL trigger with RAISE NOTICE. This lets you correlate client-side activity in django against server-side activity in PostgreSQL.
In the spirit of "Test in Production" espoused by Charity Majors, send every request to a sandbox copy of your Django app that reads/writes a sandboxed copy of your production database. In the sandbox database, raise the exception and log your traceback.
You can take this idea further and create smaller "async" setups. For example, you can, for each request, trigger a async duplicate (say, with celery) of the same request that hits a DB configured with your PostgreSQL trigger to fail and log the traceback.
Use RAISE EXCEPTION in the PostgreSQL trigger to rollback the current transaction. In Python, catch that specific exception, log it, then repeat the transaction, changing the data slightly (extra column?) to indicate that this is a retry and the trigger should not fail.
Is there a reason you can't SELECT all row values into Python, then do the detection in Python entirely?
So if you're able to detect the condition after the queries execute, then you can log the condition and/or throw an exception.
Then what you need is tooling like Sentry or New Relic.
You could use LISTEN+NOTIFY.
First let some daemon thread LISTEN and in the db trigger you can execute a NOTIFY.
The daemon thread receives the notify event and can dump the stacktrace of the main thread.
If you use psycopg2, you can use this
# Overwriting connetion.notices via Django
class MyAppConfig(AppConfig):
def ready(self):
connection_created.connect(connection_created_check_for_notice_in_connection)
class ConnectionNoticeList(object):
def append(self, message):
if not 'some_magic_of_db_trigger' in message:
return
logger.warn('%s %s' % (message, ''.join(traceback.format_stack())))
def connection_created_check_for_notice_in_connection(sender, connection, **kwargs):
connection.connection.notices=ConnectionNoticeList()
Inside an web application ( Pyramid ) I create certain objects on POST which need some work done on them ( mainly fetching something from the web ). These objects are persisted to a PostgreSQL database with the help of SQLAlchemy. Since these tasks can take a while it is not done inside the request handler but rather offloaded to a daemon process on a different host. When the object is created I take it's ID ( which is a client side generated UUID ) and send it via ZeroMQ to the daemon process. The daemon receives the ID, and fetches the object from the database, does it's work and writes the result to the database.
Problem: The daemon can receive the ID before it's creating transaction is committed. Since we are using pyramid_tm, all database transactions are committed when the request handler returns without an error and I would rather like to leave it this way. On my dev system everything runs on the same box, so ZeroMQ is lightning fast. On the production system this is most likely not an issue since web application and daemon run on different hosts but I don't want to count on this.
This problem only recently manifested itself since we previously used MongoDB with a write_convern of 2. Having only two database servers the write on the entity always blocked the web-request until the entity was persisted ( which is obviously is not the greatest idea ).
Has anyone run into a similar problem?
How did you solve it?
I see multiple possible solutions, but most of them don't satisfy me:
Flushing the transaction manually before triggering the ZMQ message. However, I currently use SQLAlchemy after_created event to trigger it and this is really nice since it decouples this process completely and thus eliminating the risk of "forgetting" to tell the daemon to work. Also think that I still would need a READ UNCOMMITTED isolation level on the daemon side, is this correct?
Adding a timestamp to the ZMQ message, causing the worker thread that received the message, to wait before processing the object. This obviously limits the throughput.
Dish ZMQ completely and simply poll the database. Noooo!
I would just use PostgreSQL's LISTEN and NOTIFY functionality. The worker can connect to the SQL server (which it already has to do), and issue the appropriate LISTEN. PostgreSQL would then let it know when relevant transactions finished. You trigger for generating the notifications in the SQL server could probably even send the entire row in the payload, so the worker doesn't even have to request anything:
CREATE OR REPLACE FUNCTION magic_notifier() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('stuffdone', row_to_json(new)::text);
RETURN new;
END;
$$ LANGUAGE plpgsql;
With that, right as soon as it knows there is work to do, it has the necessary information, so it can begin work without another round-trip.
This comes close to your second solution:
Create a buffer, drop the ids from your zeromq messages in there and let you worker poll regularly this id-pool. If it fails retrieving an object for the id from the database, let the id sit in the pool until the next poll, else remove the id from the pool.
You have to deal somehow with the asynchronous behaviour of your system. When the ids arrive constantly before persisting the object in the database, it doesnt matter whether pooling the ids (and re-polling the the same id) reduces throughput, because the bottleneck is earlier.
An upside is, you could run multiple frontends in front of this.