I'm setting up a django model to store regions, like USA, Germany, etc. I made the region name unique for the table. I have a script that populates the database from a list and if there is a duplicate region name IntegrityError is thrown as expected but then another error happens and I can't tell why from the error message. Any ideas? Thanks!
django.db.utils.DatabaseError: current transaction is aborted, commands ignored until end of transaction block
Model:
class Region(models.Model):
name = models.CharField(max_length=512, unique=True)
def __unicode__(self):
return self.name
Populate code:
try:
Region(name=server['locale']).save()
print 'Added region: %(locale)s' % server
except IntegrityError:
pass
I've confirmed that the IntegrityError is occuring but then I get this error which I dont expect:
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/models/base.py", line 456, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/models/base.py", line 549, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/models/manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/models/query.py", line 1518, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/models/sql/compiler.py", line 788, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/models/sql/compiler.py", line 732, in execute_sql
cursor.execute(sql, params)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/backends/util.py", line 15, in execute
return self.cursor.execute(sql, params)
File "/home/daedalus/webapps/wowstatus/lib/python2.6/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
django.db.utils.DatabaseError: current transaction is aborted, commands ignored until end of transaction block
You should reset your db state if something fails
for example:
from django.db import transaction
#transaction.commit_manually
def Populate():
try:
Region(name=server['locale']).save()
print 'Added region: %(locale)s' % server
except IntegrityError:
transaction.rollback()
else:
transaction.commit()
I get this error sometimes when accessing the database from the django shell. I fix it by closing the connection:
from django.db import connection
connection.close()
cur = connection.cursor()
sql = 'select distinct category from uploads_document'
cur.execute(sql)
after I have made an error like 'select ditsinct ..'
Related
I want to establish a connection with a postgres database in django. As it doesn't provide support for postgres schemas I am trying to set the search_path immediately after establishing a connection. To achieve this I have subclassed the DatabaseWrapper class and overridden the _cursor method as below:
from django.db.backends.postgresql.base import DatabaseWrapper
class DatabaseWrapper(DatabaseWrapper):
def __init__(self, *args, **kwargs):
super(DatabaseWrapper, self).__init__(*args, **kwargs)
def _cursor(self, name=None):
cursor = super(DatabaseWrapper, self)._cursor(name)
cursor.execute('SET search_path = schema_name')
return cursor
Now the above code works fine for the application code that we have written but when I try access the detail screen of any object in the admin panel of django, I get the below error trace:
File "/Users/azharuddin.syed/Desktop/application/custom_db_engine/base.py", line 13, in \_cursor
cursor.execute('SET search_path = schema_name')
File "/Users/azharuddin.syed/Desktop/application/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
return super().execute(sql, params)
File "/Users/azharuddin.syed/Desktop/application/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
return self.\_execute_with_wrappers(sql, params, many=False, executor=self.\_execute)
File "/Users/azharuddin.syed/Desktop/application/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in \_execute_with_wrappers
return executor(sql, params, many, context)
File "/Users/azharuddin.syed/Desktop/application/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in \_execute
return self.cursor.execute(sql, params)
File "/Users/azharuddin.syed/Desktop/application/venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/azharuddin.syed/Desktop/application/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 82, in \_execute
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: syntax error at or near "SET"
LINE 1: ...6171701248_sync_1" NO SCROLL CURSOR WITH HOLD FOR SET search...
If I understand correctly then two queries are getting mixed together and getting executed at the same time. Why is this the case?
P.S: I am aware that we can pass options parameter with the search path while defining the connections in django but this is not working in my case as the DB is behind a proxy. Any other solutions will be welcomed.
In my Django project, I have a view that when a user posts a zip file, it will respond immediately back and then process the data in the background with the help of threading. The view works fine in normal test, but when I run the Django's test it fails with a database table is locked error. Currently, I'm using default SQLite database and I know if I switch to another database this problem may be solved but I'm seeking an answer for the current setup. I trimmed the code for simplicity.
It seems that the problem is with writing in DeviceReportModel table. But I'm not sure why TestDeviceReport accessing it.
Model.py:
class DeviceReportModel(models.Model):
device_id = models.PositiveIntegerField(primary_key=True)
ip = models.GenericIPAddressField()
created_time = models.DateTimeField(default=timezone.now)
report_file = models.FileField(upload_to="DeviceReport")
device_datas = models.ManyToManyField(DeviceDataReportModel)
def __str__(self):
return str(self.id)
Serializers.py:
class DeviceReportSerializer(serializers.ModelSerializer):
class Meta:
model = DeviceReportModel
fields = '__all__'
read_only_fields = ('created_time', 'ip', 'device_datas')
views.py:
from django.utils import timezone
from django.core.files.base import ContentFile
from rest_framework.response import Response
from rest_framework import status, generics
import time
import threading
from queue import Queue
class DeviceReportHandler:
ReportQueue = Queue()
#staticmethod
def save_datas(device_object, request_ip, b64datas):
device_data_models = []
# ...
# process device_data_models
# this will take some time
time.sleep(10)
return device_data_models
#classmethod
def Check(cls):
while(True):
if not cls.ReportQueue.empty():
report = cls.ReportQueue.get()
# ...
report_model = DeviceReportModel(
device_id=report['device_object'], ip=report['request_ip'])
# THIS LINE GIVES ERROR
report_model.report_file.save(
"Report_{}.txt.gz".format(timezone.now()), ContentFile(report['report_data']))
device_data_models = cls.save_datas(
report['device_object'], report['request_ip'], 'SomeData')
report_model.device_datas.set(device_data_models)
report_model.save()
print("Report Handle Done")
time.sleep(.1)
#classmethod
def run(cls):
thr = threading.Thread(target=cls.Check)
thr.daemon = True
thr.start()
class DeviceReportView(generics.ListCreateAPIView):
queryset = DeviceReportModel.objects.all()
serializer_class = DeviceReportSerializer
DeviceReportHandler.run()
def post(self, request):
# ...
report = {
'device_object': 1,
'request_ip': '0.0.0.0',
'report_data': b'Some report plain data',
}
# add request to ReportQueue
DeviceReportHandler.ReportQueue.put(report)
return Response("OK", status.HTTP_201_CREATED)
tests.py:
from rest_framework.test import APITestCase
import gzip
from io import BytesIO
import base64
import time
class TestDeviceReport(APITestCase):
#classmethod
def setUpTestData(cls):
# add a new test device for other tests
pass
def generate_device_data(self):
# generate fake device data
return ""
def test_Report(self):
# generate device data
device_data = ''
for i in range(10):
device_data += self.generate_device_data() + '\n'
buf = BytesIO()
compressed = gzip.GzipFile(fileobj=buf, mode="wb")
compressed.write(device_data.encode())
compressed.close()
b64data = base64.b64encode(buf.getvalue()).decode()
data = {
"device_id": 1,
"report_data": b64data
}
response = self.client.post(
'/device/reports/', data=data, format='json')
print(response.status_code, response.content)
def tearDown(self):
# put some sleep to check whether the data has been processed
# see "Report Handle Done"
time.sleep(10)
And here is error log:
(myDjangoEnv) python manage.py test deviceApp.tests.tests.TestDeviceReport
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
201 b'"OK"'
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\sqlite3\base.py", line 383, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database table is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "<project_path>\deviceApp\views.py", line 303, in Check
"Report_{}.txt.gz".format(timezone.now()), ContentFile(report['report_data']))
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\fields\files.py", line 93, in save
self.instance.save()
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\base.py", line 741, in save
force_update=force_update, update_fields=update_fields)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\base.py", line 779, in save_base
force_update, using, update_fields,
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\base.py", line 870, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\base.py", line 908, in _do_insert
using=using, raw=raw)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\models\sql\compiler.py", line 1335, in execute_sql
cursor.execute(sql, params)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\Masoud\Anaconda3\envs\myDjangoEnv\lib\site-packages\django\db\backends\sqlite3\base.py", line 383, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database table is locked
.
----------------------------------------------------------------------
Ran 1 test in 10.023s
OK
Destroying test database for alias 'default'...
Database is locked errors
SQLite is meant to be a lightweight database, and thus can’t support a high level of concurrency. OperationalError: database is locked errors indicate that your application is experiencing more concurrency than sqlite can handle in default configuration. This error means that one thread or process has an exclusive lock on the database connection and another thread timed out waiting for the lock the be released.
Python’s SQLite wrapper has a default timeout value that determines how long the second thread is allowed to wait on the lock before it times out and raises the OperationalError: database is locked error.
If you’re getting this error, you can solve it by:
Switching to another database backend. At a certain point SQLite becomes too “lite” for real-world applications, and these sorts of concurrency errors indicate you’ve reached that point.
Rewriting your code to reduce concurrency and ensure that database transactions are short-lived.
Increase the default timeout value by setting the timeout database option:
'OPTIONS': {
# ...
'timeout': 20,
# ...
}
This will make SQLite wait a bit longer before throwing “database is locked” errors; it won’t really do anything to solve them.
https://docs.djangoproject.com/en/3.0/ref/databases/#database-is-locked-errorsoption
Try using django.test.TransactionTestCase instead of TestCase
I am trying to create a simple model for my existing database with peewee.
The database that I'm working on is described in this paper.
I've generated a model with pwiz which works fine, however it didn't create foreign keys for me which enable joining. Therefore I edited the Model so it looks like this:
from peewee import *
database = MySQLDatabase('enron', **{'password': '...', 'user': 'root'})
class UnknownField(object):
pass
class BaseModel(Model):
class Meta:
database = database
class Employee(BaseModel):
email = CharField(db_column='Email_id', unique=True)
eid = PrimaryKeyField()
class Meta:
db_table = 'employeelist'
class Message(BaseModel):
mid = PrimaryKeyField()
sender = ForeignKeyField(Employee,
related_name='messages',
to_field='email') # was CharField() before my edit
class Meta:
db_table = 'message'
However when I try to run:
for message in Message.select():
print message.mid
I get an error:
Traceback (most recent call last):
File "DBModelEnron.py", line 62, in <module>
for message in Message.select():
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2514, in __iter__
return iter(self.execute())
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2507, in execute
self._qr = ResultWrapper(model_class, self._execute(), query_meta)
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2203, in _execute
return self.database.execute_sql(sql, params, self.require_commit)
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2833, in execute_sql
self.commit()
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2688, in __exit__
reraise(new_type, new_type(*exc_value.args), traceback)
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2825, in execute_sql
cursor.execute(sql, params or ())
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
peewee.OperationalError: (1054, "Unknown column 't1.sender_id' in 'field list'")
I found a similar problem here, however I have a proper PrimaryKey defined.
Just add a db_column='' to your foreign keys:
sender = ForeignKeyField(Employee,
db_column='sender', # Added this.
related_name='messages',
to_field='email') # was CharField() before my edit
I'm running into a DB related problem in Django that I don't understand.
I define an MPTT model like so:
class Image(MPTTModel):
name = models.CharField(max_length=50)
parent = TreeForeignKey('self', null=True, blank=True, related_name='children')
def __unicode__(self):
return self.name
def rank(self): leaves = self.get_leafnodes() if leaves: rank = leaves[0].get_level() - self.get_level() else: rank = 0 return rank
mptt.register(Image, order_insertion_by=['name'])
Then in my views, I attempt a few statements with the model, and I get an OperationalError.
def index(request):
if request.method == 'POST':
image_string = request.POST.get('get_image')
index = image_string.find('(')
if index == -1:
parent = image_string
child = None
else:
parent = image_string[0:index]
child = image_string[index+1:len(image_string)-1]
try:
images = Image.objects.all()
image_names = [a.name for a in images]
except Image.DoesNotExist:
return render(request, 'images_app/index.html', {'images':[]})
else:
parent_model = Image(name=parent)
parent_model.save()
child_model = Image(name=child, parent=parent_model)
child_model.save()
return render(request, 'images_app/index.html', {'images':images})
I'm not sure if this is a problem with my view or the way I'm defining the model. According to my understanding, the 'try' expression should make it so that if the code doesn't evaluate, it will simply skip to the exception. Why doesn't this go straight to the exception?
Traceback:
File "C:\Python27\lib\site-packages\django\core\handlers\base.py" in get_response
114. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\djangoprojects\images\images_app\views.py" in index
17. if images:
File "C:\Python27\lib\site-packages\django\db\models\query.py" in __nonzero__
100. self._fetch_all()
File "C:\Python27\lib\site-packages\django\db\models\query.py" in _fetch_all
854. self._result_cache = list(self.iterator())
File "C:\Python27\lib\site-packages\django\db\models\query.py" in iterator
220. for row in compiler.results_iter():
File "C:\Python27\lib\site-packages\django\db\models\sql\compiler.py" in results_iter
709. for rows in self.execute_sql(MULTI):
File "C:\Python27\lib\site-packages\django\db\models\sql\compiler.py" in execute_sql
782. cursor.execute(sql, params)
File "C:\Python27\lib\site-packages\django\db\backends\util.py" in execute
69. return super(CursorDebugWrapper, self).execute(sql, params)
File "C:\Python27\lib\site-packages\django\db\backends\util.py" in execute
53. return self.cursor.execute(sql, params)
File "C:\Python27\lib\site-packages\django\db\utils.py" in __exit__
99. six.reraise(dj_exc_type, dj_exc_value, traceback)
File "C:\Python27\lib\site-packages\django\db\backends\util.py" in execute
53. return self.cursor.execute(sql, params)
File "C:\Python27\lib\site-packages\django\db\backends\sqlite3\base.py" in execute
450. return Database.Cursor.execute(self, query, params)
Exception Type: OperationalError at /images/
Exception Value: no such table: images_app_image
By looking at the exception value (no such table: images_app_image), I would guess that the actual database table doesn't exist.
Check if the table exists in your database by using the ./manage dbshell command. You can list all tables in the database within the shell with the command .schema or use .schema images_app_image to only show the schema definition for the actual table.
If the table doesn't exist, create it with ./manage syncdb (or use the migrate command if you are using South).
Try python manage.py syncdb.
If you get
Unknown command: 'syncdb'
You can try
python manage.py migrate --run-syncdb
I am attempting to save a parent/children set of records, and I want to wrap the inserts in a transaction. I am using SQLAlchemy with postgresql 8.4.
Here is a snippet of my code:
def insert_data(parent, child_rows):
# Start a transaction
conn = _get_connection()
tran = conn.begin()
try:
sql = get_sql_from_parent(parent)
res = conn.execute(sql) # <- Code barfs at this line
item = res.fetchone() if res else None
parent_id = item['id'] if ((item) and ('id' in item)) else -1
if parent_id == -1:
raise Exception('Parent could not be saved in database')
# Import children
for child in child_rows:
child_sql = get_child_sql(parent_id, child)
conn.execute(child_sql)
tran.commit()
except IntegrityError:
pass # rollback?
except Exception as e:
tran.rollback()
print "Exception in user code:"
print '-'*60
traceback.print_exc(file=sys.stdout)
print '-'*60
When I invoke the function, I get the following stacktrace:
Traceback (most recent call last):
File "import_data.py", line 125, in <module>
res = conn.execute(sql)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.7.4-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1405, in execute
params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.7.4-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1582, in _execute_text
statement, parameters
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.7.4-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1646, in _execute_context
context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.7.4-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1639, in _execute_context
context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.7.4-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 330, in do_execute
cursor.execute(statement, parameters)
InternalError: (InternalError) current transaction is aborted, commands ignored until end of transaction block
...
Does anyone know why I am getting this cryptic error - and how do I resolve it?
Can you activate the log query on postgresql ? (min_duration set to 0 in postgresql.conf then restart).
Then look at your postgresql logs to debug it.