Develop Reference

Django Migration Error with MySQL: BLOB/TEXT column 'id' used in key specification without a key length"

We have Django Model, use Binary Field for ID.
# Create your models here.
class Company(models.Model):
id = models.BinaryField(max_length=16, primary_key=True)
name = models.CharField(max_length=12)
class Meta:
db_table = "company"
We use MySQL Database and have error when migrate.
File "/home/cuongtran/Downloads/sample/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 270, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (1170, "BLOB/TEXT column 'id' used in key specification without a key length")
Do you have any solution? We need to use MySQL and want to use the Binary Field for ID.
Thank you!

I think you cannot achieve this. Based on Django documentation it looks like use of binary fields is discouraged
A field to store raw binary data. It only supports bytes assignment.
Be aware that this field has limited functionality. For example, it is
not possible to filter a queryset on a BinaryField value. It is also
not possible to include a BinaryField in a ModelForm.
Abusing BinaryField
Although you might think about storing files in the database, consider
that it is bad design in 99% of the cases. This field is not a
replacement for proper static files handling.
And based on a Django bug, it is most likely impossible to achieve a unique value restriction on a binary field. This bug is marked as wont-fix. I am saying most likely impossible as I did not find evidence to confirm that binary field is stored as a BLOB field but the error does allude to it.
Description
When I used a field like this:
text = models.TextField(maxlength=2048, unique=True)
it results in the following sql error when the admin app goes to make the table
_mysql_exceptions.OperationalError: (1170, "BLOB/TEXT column 'text' used in key specification without a key length")
After a bit of investigation, it turns out that mysql refuses to use unique with the column unless it is only for an indexed part of the text field:
CREATE TABLE `quotes` ( \`id\` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `text` longtext NOT NULL , \`submitTS\` datetime NOT NULL, `submitIP` char(15) NOT NULL, `approved` bool NOT NULL, unique (text(1000)));
Of course 1000 is just an arbitrary number I chose, it happens to be the maximum my database would allow. Not entirely sure how this can be fixed, but I figured it was worth mentioning.

MySQL restricts the primary key on BLOB/TEXT column to first N chars, when you generates migration file using Django's makemigrations command, BinaryField in Django is mapped to longblob which is BLOB column in MySQL without specifying the key length.
Which means your Django model definition :
class Company(models.Model):
id = models.BinaryField(max_length=16, primary_key=True)
name = models.CharField(max_length=12)
class Meta:
db_table = "company"
will be converted to SQL expression that causes this error (You can check out the detailed SQL expressions by sqlmigrate command) :
CREATE TABLE `company` (`id` longblob NOT NULL PRIMARY KEY,
`name` varchar(12) NOT NULL);
while the correct SQL expression for MySQL should be like this :
CREATE TABLE `company` (`id` longblob NOT NULL,
`name` varchar(12) NOT NULL);
ALTER TABLE `company` ADD PRIMARY KEY (id(16));
where PRIMARY KEY (id(16)) comes from your id length in the BLOB column, used to structure primary key index of the table.
So the easiest solution is as described in the accepted answer -- avoid BinaryField in Django as primary key, or you can manually add raw SQL scripts to your migration file if you really need BinaryField (BLOB column) to be primary key and you are sure the id field will NOT go beyond the specific size (in your case, 16 bytes).

Django, Postgres - column cannot be cast automatically to type integer

In my database i have column:
currency = models.CharField(max_length=10, blank=True, null=True)
I want to change this column from CharField to IntegerField. So in models.py i change this:
currency = models.IntegerField(blank=True, null=True)
then i made migrations: python manage.py makemigrations and python manage.py migrate. After that actions it rise error:
django.db.utils.ProgrammingError: column "currency" cannot be cast automatically to type integer
HINT: Specify a USING expression to perform the conversion.
After that in pgAdmin3 console i made this changes:
ALTER TABLE my_table ALTER COLUMN currency TYPE integer USING (currency::integer);
But i still got that error, I tried to change all back, but error doesn't disappear. What i have to do to escape this error. Thank you

I think django migrations does not perform casting, I looked in the documentation but I did not find any thing about column casting.
if the existing data is not that important for you, you can delete the column and create a new one
first step remove currency from you model and apply migration
add again the currency with the new definition and apply again the migration
if you want to keep your data, you need to give your new column a different name and use the old column as a temporary column to hold the data during the transition.
Important: Postgresql is more strongly typed in recent versions, and as explained here some casting may not work in PosgreSQL unless it's explicitly done. And it required to be more specific about the type. So you have to make the right choice based on your values:
alter table my_table alter column currency type bigint using currency::bigint
or maybe:
alter table my_table alter column currency type numeric(10,0) using currency::numeric

It is a PSQL issue when changing from certain data types to others... I had a similar problem an I did something a bit hackey but it worked ... but only because I didn't have any important data in the column
1) delete the most recent migration for that app
2) delete/comment out the "column" on the object
3) migrate the app with the missing column
4) reinstate/uncomment the offending "column"
5) migrate again
this is all a long way to delete and recreate the column on the actual db without using sql ... figured I would share in case it might help someone down the road

How do I fix django.db.utils.IntegrityError: duplicate key value violates unique constraint?

I'm getting the following error:
django.db.utils.IntegrityError: duplicate key value violates unique constraint "record_coordinates_lat_lon_created_by_id_key"
DETAIL: Key (lat, lon, created_by_id)=(34.84000015258789, -111.80000305175781, 2) already exists.
Background: I've been using MySQL with Django 1.4.3 up until this point. Now I've installed Postgres 9.3 and Psycopg2 2.5.2. Validate and Syncdb worked fine. South is NOT installed.
I ran a script (which works with MySQL). The script loops through GPS files and saves lat/lon data in a coordinates table.
_coordinates, coordinates_created = Coordinates.objects.get_or_create(
lat=round(group[u'lat'], Coordinates.max_decimal_places),
lon=round(group[u'lon'], Coordinates.max_decimal_places),
created_by=self._user,
modified_by=self._user,
date_created=datetime.now(), # See Update 2 addition below.
)
I have a unique_together = ('lat', 'lon', ) constraint in the model definition. Some of the coordinates are identical (thus the use of get_or_create()). I find myself scratching my head because it should be 'getting' the coordinates and not attempting to 'create' new ones.
Most of the questions on this site about Postgres and Django almost inevitably mention South. Do I need South, or is something else going on here? I just wanted to do a quick and dirty test without installing migrations.
Update 1:
Something else I tried was to run: SELECT setval('django_content_type_id_seq', (SELECT MAX(id) FROM django_content_type)); in Postgres on the suggestion of another post. The error persists.
Update 2:
Okay I didn't realize that I needed to put all of my Coordinates fields in the defaults dict. The Coordinates model contains another field 'date_created=models.DateTimeField(auto_now_add=True)'
I found the following blog post that seems to explain that get_or_create() breaks when you use 'auto_now_add=True'. The big question is now how do I use auto_now_add and without breaking get_or_create()?

This answered my question.
_coordinates, coordinates_created = Coordinates.objects.get_or_create(
lat=Decimal(group[u'lat'])._rescale(-Coordinates.max_decimal_places, 'ROUND_HALF_EVEN'),
lon=Decimal(group[u'lon'])._rescale(-Coordinates.max_decimal_places, 'ROUND_HALF_EVEN'),
created_by=self._user,
modified_by=self._user,
)
The auto_now and auto_now_add were just fine. It turns out that all of my defaults are already defined on the Model.
The problem was that group[u'lat'] and group[u'lon'] were both cast as floats when I placed them in my dictionary. In contrast, lat and lon are both defined as DecimalFields().
When using MySQL I can compare these float values against the contents of the database just fine. However, when I use Postgres the get() portion of get_or_create() attempts to compare a Decimal value from the database with the the float values I was providing it. The types are interpreted more strongly, and the float will not be cast to a Decimal during the comparison.
In my debugger, I saw:
{Decimal}lat
{float}group[lat]
It would be nice if django could produce a clear error like TypeError: Can't compare Decimal with float.

You missed defaults argument in the
get_or_create call. Then, if there is no record in the db with the specified lat, lon, it will create a new record with the default lat, lon, which obviously are not autogenerated and you get the IntegrityError.
_coordinates, coordinates_created = Coordinates.objects.get_or_create(
lat=round(group[u'lat'], Coordinates.max_decimal_places),
lon=round(group[u'lon'], Coordinates.max_decimal_places),
created_by=self._user,
defaults=dict(
lat=round(group[u'lat'], Coordinates.max_decimal_places),
lon=round(group[u'lon'], Coordinates.max_decimal_places),
created_by=self._user,
modified_by=self._user,
)
)
As the unique index consist of columns lat, lon and created_by (looking at the index name from the error), you should use all of them in the filters of get_or_create.

Django get_or_create raises Duplicate entry for key Primary with defaults

Help! Can't figure this out! I'm getting a Integrity error on get_or_create even with a defaults parameter set.
Here's how the model looks stripped down.
class Example(models.Model):model
user = models.ForeignKey(User)
text = models.TextField()
def __unicode__(self):
return "Example"
I run this in Django:
def create_example_model(user, textJson):
defaults = {text: textJson.get("text", "undefined")}
model, created = models.Example.objects.get_or_create(
user=user,
id=textJson.get("id", None),
defaults=defaults)
if not created:
model.text = textJson.get("text", "undefined")
model.save()
return model
I'm getting an error on the get_or_create line:
IntegrityError: (1062, "Duplicate entry '3020' for key 'PRIMARY'")
It's live so I can't really tell what the input is.
Help? There's actually a defaults set, so it's not like, this problem where they do not have a defaults. Plus it doesn't have together-unique. Django : get_or_create Raises duplicate entry with together_unique
I'm using python 2.6, and mysql.

You shouldn't be setting the id for objects in general, you have to be careful when doing that.
Have you checked to see the value for 'id' that you are putting into the database?
If that doesn't fix your issue then it may be a database issue, for PostgreSQL there is a special sequence used to increment the ID's and sometimes this does not get incremented. Something like the following:
SELECT setval('tablename_id_seq', (SELECT MAX(id) + 1 FROM
tablename_id_seq));

get_or_create() will try to create a new object if it can't find one that is an exact match to the arguments you pass in.
So is what I'm assuming is happening is that a different user has made an object with the id of 3020. Since there is no object with the user/id combo you're requesting, it tries to make a new object with that combo, but fails because a different user has already created an item with the id of 3020.
Hopefully that makes sense. See what the following returns. Might give a little insight as to what has gone on.
models.Example.objects.get(id=3020)
You might need to make 3020 a string in the lookup. I'm assuming a string is coming back from your textJson.get() method.

One common but little documented cause for get_or_create() fails is corrupted database indexes.
Django depends on the assumption that there is only one record for given identifier, and this is in turn enforced using UNIQUE index on this particular field in the database. But indexes are constantly being rewritten and they may get corrupted e.g. when the database crashes unexpectedly. In such case the index may no longer return information about an existing record, another record with the same field is added, and as result you'll be hitting the IntegrityError each time you try to get or create this particular record.
The solution is, at least in PostgreSQL, to REINDEX this particular index, but you first need to get rid of the duplicate rows programmatically.

AppEngine: Query datastore for records with <missing> value

I created a new property for my db model in the Google App Engine Datastore.
Old:
class Logo(db.Model):
name = db.StringProperty()
image = db.BlobProperty()
New:
class Logo(db.Model):
name = db.StringProperty()
image = db.BlobProperty()
is_approved = db.BooleanProperty(default=False)
How to query for the Logo records, which to not have the 'is_approved' value set?
I tried
logos.filter("is_approved = ", None)
but it didn't work.
In the Data Viewer the new field values are displayed as .

According to the App Engine documentation on Queries and Indexes, there is a distinction between entities that have no value for a property, and those that have a null value for it; and "Entities Without a Filtered Property Are Never Returned by a Query." So it is not possible to write a query for these old records.
A useful article is Updating Your Model's Schema, which says that the only currently-supported way to find entities missing some property is to examine all of them. The article has example code showing how to cycle through a large set of entities and update them.

A practice which helps us is to assign a "version" field on every Kind. This version is set on every record initially to 1. If a need like this comes up (to populate a new or existing field in a large dataset), the version field allows iteration through all the records containing "version = 1". By iterating through, setting either a "null" or another initial value to the new field, bump the version to 2, store the record, allows populating the new or existing field with a default value.
The benefit to the "version" field is that the selection process can continue to select against that lower version number (initially set to 1) over as many sessions or as much time is needed until ALL records are updated with the new field default value.

Maybe this has changed, but I am able to filter records based on null fields.
When I try the GQL query SELECT * FROM Contact WHERE demo=NULL, it returns only records for which the demo field is missing.
According to the doc http://code.google.com/appengine/docs/python/datastore/gqlreference.html:
The right-hand side of a comparison can be one of the following (as
appropriate for the property's data type): [...] a Boolean literal, as TRUE or
FALSE; the NULL literal, which represents the null value (None in
Python).
I'm not sure that "null" is the same as "missing" though : in my case, these fields already existed in my model but were not populated on creation. Maybe Federico you could let us know if the NULL query works in your specific case?