I'm getting an OperationalError: (OperationalError) database is locked error when connection via SQLAlchemy in an IPython notebook instance and I'm not sure why.
I've written a Python interface to a SQLite database using SQLAlchemy and the Declarative Base syntax. I import the database models into an IPython notebook to explore the data. This worked just fine this morning. Here is the code:
from psf_database_interface import session, PSFTable
query = session.query(PSFTable).first()
But this afternoon after I closed my laptop with IPython running (it restarts the server just fine) I started getting this error. It's strange because I can still open the database from the SQLite3 command line tool and query data. I don't expect any other processes to be connecting to this database and running fuser on the database confirms this. My application is not using any concurrent processes (in the code I've written, IDK if something is buried in SQLAlchemy or IPython), and even if it were I'm just doing a read operation, which SQLite does support concurrently.
I've tried restarting the IPython kernel as well as killing and restarting the IPython notebook server. I've tried creating a backup of the database and replacing the database with the backup as suggested here: https://stackoverflow.com/a/2741015/1216837. Lastly, out of desperation, I tried adding the following to see if I could clean out something stuck in the session somehow:
print session.is_active
session.flush()
session.close()
session.close_all()
print session.is_active
Which returns True and True. Any ideas?
Update: I can run the code snippet that is causing errors from a python file without any issues, the issue only occurs in IPython.
I faced the same problem. I can run python scripts but the IPython raise the below exception.
You need to check with fuser there is no process which is using this. But if you cannot find anything and your history of commands are not important to you, you can use the following workaround.
When I deleted the /home/my_user/.ipython/profile_default/history.sqlite file, I can start the IPython. The history is empty as I mentioned above.
$ ipython
[TerminalIPythonApp] ERROR | Failed to create history session in /home/my_user/.ipython/profile_default/history.sqlite. History will not be saved.
Traceback (most recent call last):
File "/home/esadrfa/libs/anaconda3/lib/python3.6/site-packages/IPython/core/history.py", line 543, in __init__
self.new_session()
File "<decorator-gen-22>", line 2, in new_session
File "/home/esadrfa/libs/anaconda3/lib/python3.6/site-packages/IPython/core/history.py", line 58, in needs_sqlite
return f(self, *a, **kw)
File "/home/esadrfa/libs/anaconda3/lib/python3.6/site-packages/IPython/core/history.py", line 570, in new_session
self.session_number = cur.lastrowid
sqlite3.OperationalError: database is locked
[TerminalIPythonApp] ERROR | Failed to open SQLite history :memory: (database is locked).
Related
Context:
We run Cypress tests which use instances of our application started using manage.py test package.test.suite. The test fixtures and environment are all set up using a class extended from django.contrib.staticfiles.testing.StaticLiveServerTestCase. A unit test/method is added to the extended class which invokes Cypress as a subprocess to run the tests and asserts against the exit code of the subprocess.
Versions:
Python: 3.6.8
Django: 2.2.3
macOS: Mojave 10.14.6
The problem:
This worked well until yesterday when I updated Cypress via npm. Now when I start a server using manage.py test some.test.suite the server will fail to serve all of the static resources requested by the browser. Specifically, it almost always fails to serve some .woff fonts and a random javascript/css file or two. The rest of the files are served but those 5 or 6 which the browser never receives a response for. Eventually I'll get a ConnectionResetError: [Errno 54] Connection reset by peer error (stack trace below) in the terminal. Additionally, if I enable the cache in my browser and attempt a few refreshes things will work fine, almost as if theres a limit to the number of files that can be served at once and once some files are cached in the browser the number of requested files falls below that limit.
When I do python manage.py runserver, however, I don't seem to have this problem at all with or without caching enabled.
Stack Trace:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 52655)
Traceback (most recent call last):
File "/Users/myuser/.pyenv/versions/3.6.8/lib/python3.6/socketserver.py", line 320, in _handle_request_noblock
self.process_request(request, client_address)
File "/Users/myuser/.pyenv/versions/3.6.8/lib/python3.6/socketserver.py", line 351, in process_request
self.finish_request(request, client_address)
File "/Users/myuser/.pyenv/versions/3.6.8/lib/python3.6/socketserver.py", line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/Users/myuser/.pyenv/versions/3.6.8/lib/python3.6/socketserver.py", line 724, in __init__
self.handle()
File "/Users/myuser/Path/To/Project/venv/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 171, in handle
self.handle_one_request()
File "/Users/myuser/Path/To/Project/venv/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 179, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/Users/myuser/.pyenv/versions/3.6.8/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
----------------------------------------
Additional Notes/Observations
The problem also occurs when I run the tests headless.
Installing Cypress did run into a hitch that required me to reinstall Electron and Cypress, clear the npm cache and delete the Cypress cache folder.
It's also strange that the problem seems to be with the python environment and not Cypress itself since accessing the web server via my browser and not Cypress produces the same behaviour; so perhaps Cypress is just a red herring and some sort of shared dependency was changed/updated in the process.
If I attempt to directly access one of the files that doesn't get served during a request, which is displayed as 'pending' in Chrome Developer's tools network tab, directly via a second tab it will succeed (but sometimes be lagged by several seconds).
After closing the test server and attempting to run it again I will get an [OSError: [Errno 48] Address already in use] error for up to a minute or two. Previously the server would relinquish the address/port immediately upon closing (or I'm assuming as such since I've never seen this before and had rapidly closed and re-ran tests to test fixture changes in the past).
Things I've Tried:
Rebuilding my virtualenv from scratch
Copying my old venv folder from a Time Machine backup from a time when things worked
Reverting the version of Cypress back to what it was prior to the problem
Looking into timeouts and connection limits of using manage.py test vs manage.py runserver (Didn't find anything).
Toggling DEBUG mode and profiling on/off.
Switching between Chrome and Chromium in Cypress
Building new environments with different python versions (3.6.5, 3.6.7 and 3.6.8)
Switching Django back to 2.2.2 from 2.2.3
Any help would be appreciated!
Update
Looking into this further it looks like the requests for the files that don't get a response never make it to the WSGIHandler form the Socket layer (or even if they make it to the socket layer, I'd assume they do though).
Update 2
I see the same behaviour with manage.py runserver if I include the --nothreading switch. I had a co-worker give it a test and he indeed saw the same behaviour with manage.py runserver --nothreading but manage.py test test.to.run still functioned fine for him.
Also removing the font references from the css/templates just results in a different set of 5 files that aren't served.
After following the steps in the official wiki I keep getting the following error when launching with breakpoints or setting breakpoints:
/ptvsd/wrapper.py", line 423, in pydevd_request
os.write(self.pipe_w, s.encode('utf8'))
File "google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/stubs.py", line 40, in os_error_not_implemented
raise OSError(errno.ENOSYS, 'Function not implemented')
OSError: [Errno 38] Function not implemented
The application runs anyway but the breakpoints are never hit. It seems that ptvsd is trying to use some method that is blocked by the app engine sandboxed environment. I'm running vscode in a python virtualenv, any clue?
My solution was to use PyCharm community edition's debugger, its similar perhaps more capable IDE and debugger for Python specific debugging.
I have tried to find a reliable way to get rid of this error but it's proving quite difficult. Here are some advices though:
Use the --threadsafe_override=default:false flag when running the app engine dev server as explained here.
The app engine dev server must be launched from vscode(for example via a task) instead of a separate terminal window.
If you still get the error, stop the debugger, kill the task and restart everything.
(After that the debugger correctly hits the breakpoints, but curiously the callstack is set to the main thread instead of the thread containing the breakpoint, you need to manually click on the correct thread in the callstack window.)
I have a flask app that creates a sqlite db to load fixtures for tests. When I run pytest within osx, there are no issues. However, when I set 'PRAGMA journal_mode=WAL' within a ubuntu 14.04 docker container, I get this:
disk I/O error
Traceback (most recent call last):
File "/tmp/my_app/util/sqlalchemy_helpers.py", line 23, in pragma_journalmode_wal
cursor.execute('PRAGMA journal_mode=WAL')
OperationalError: disk I/O error
The sqlite db file is written to a folder within tmp that is dynamically created using python's "tempfile.mkdtemp" function. Even though the tests run as root (because docker), I still made sure the folder has full read/write/execute permissions. I verified that there is plenty of space left on /tmp. I have test code that creates, modifies, and deletes a file in the database folder, and it passes successfully.
I cannot seem to find a way to get an error code or better explanation as to what failed. Any ideas how I can better debug the issue? Could there be an issue with the docker container?
I had similar problem right now, when recreating sqlite3 database:
Removed database.sqlite3
Created database.sqlite3
Setup right permissions.
Error ocurred.
After some titme figured out that I have also database.sqlite3-shm and database.sqlite3-wal
Removed database.sqlite3-shm and database.sqlite3-wal
And everything goes back to normal.
Periodically when using pyodbc to create a connection to an oracle database, it kills the interpreter.
import pyodbc
connectString = 'Driver={Microdsoft ODBC for Oracle};Server=<host>:<port>/<db>.<host>;uid=<username>;pwd=<password>'
cnxn = pyodbc.connect(connectString)
Say 1/5 of the time this will just drop me from the Python prompt (>>>) down to my dos prompt (C:)
Any ideas of why this happens or how to fix it?
We could not discover what was exactly causing this problem. Research leads us to think that there may be a problem with Python 2.7.1, PyODBC, and Win 2003 server.
To fix this problem we created a jar file that executed the permission check and then had the python script call that file and return the result
I have a Django installation running on the development server. I recently used python manage.py loaddata 8000models.json. Now everything is super slow. Pages won't load. python manage.py flush does not return. ModelType.objects.count() does not return.
(Or maybe they will return if I wait a sufficiently long time.)
What's going on here? Is Django truly unable to handle that much data? Or is there some other issue?
Update: I observe this issue on PostgreSQL but not SQLite with the same amount of data. Perhaps I just need to wipe the PostgreSQL and reload the data.
Update 2: How can I wipe the PostgreSQL and reset? python manage.py reset appname isn't responding.
Update 3: This is how I'm trying to wipe the PostgreSQL:
#! bin/bash
sudo -u postgres dropdb mydb
sudo -u postgres createdb mydb
sudo -u postgres psql mydb < ~/my-setup/my-init.sql
python ~/path/to/manage.py syncdb
However, this causes the following errors:
dropdb: database removal failed: ERROR: database "mydb" is being accessed by other users
DETAIL: There are 8 other session(s) using the database.
createdb: database creation failed: ERROR: database "mydb" already exists
ERROR: role "myrole" cannot be dropped because some objects depend on it
DETAIL: owner of table mydb.mytable_mytable
# ... more "owner of table", "owner of sequence" statements, etc
How can I close out these other sessions? I don't have Apache running. I've only been using one instance of the Django development server at a time. However, when it got unresponsive I killed it with Control+Z. Perhaps that caused it to not release a database connection, thereby causing this issue? How can I get around this?
Ctrl-Z just stops the process, it does not kill it. (I assume you're using bash) type jobs in your terminal, and you should see the old processes still running.
Once you kill all the jobs that are accessing the PostgreSQL database, you should be able to drop, create, and syncdb as you expect.
Have you tried checking that your postrgres database is still replying at all, from the sounds of it it is probably processing the data you told it to load.
When loading the data you should just let it finish first. If pages are still taking a very long time to render you should check your queries instead.
Using .count(), even in PostgreSQL (which has rather slow counts) with 8000 items should return within a good amount of time.