Develop Reference

PostgreSQL JOIN on JSON Object column

I'm supposed to join 3 different tables on postgres:
lote_item (on which I have some books id's)
lote_item_log (on which I have a column "attributes", with a JSON object such as {"aluno_id": "2823", "aluno_email": "someemail#outlook.com", "aluno_unidade": 174, "livro_codigo": "XOZK-0NOYP0Z1EMJ"}) - Obs.: Some values on aluno_unidade are null
and finally
company (on which I have every school name for every aluno_unidade.
Ex: aluno_unidade = 174 ==> nome_fantasia = mySchoolName).
Joining the first two tables was easy, since lote_item_log has a foreign key which I could match like this:
SELECT * FROM lote_item JOIN lote_item_log ON lote_item.id = lote_item_log.lote_item_id
Now, I need to get the School Name, contained on table company, with the aluno_unidade ID from table lote_item_log.
My current query is:
SELECT
*
FROM
lote_item
JOIN
lote_item_log
ON
lote_item.id = lote_item_log.lote_item_id
JOIN
company
ON
(
SELECT
JSON_EXTRACT_PATH_TEXT(attributes, 'aluno_unidade')::int
FROM
lote_item_log
WHERE
operation_id = 6
) = company.senior_id
WHERE
item_id = {book_id};
operation_id determines which school is active.
ERROR I'M GETTING:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.CardinalityViolation) more than one row returned by a subquery used as an expression
I tried LIMIT 1, but then I got just an empty array.
What I need is:
lote_item.created_at | lote_item.updated_at | lote_item.item_id | uuid | aluno_email | c014_id | nome_fantasia | cnpj | is_franchise | is_active
somedate | somedate | some_item_id | XJW4 | someemail#a | some_id | SCHOOL NAME | cnpj | t | t

I got it.
Not sure it's the best way, but worked...
SELECT
*
FROM
lote_item
JOIN
lote_item_log
ON
lote_item.id = lote_item_log.lote_item_id
JOIN
company
ON
JSON_EXTRACT_PATH_TEXT(attributes, 'aluno_unidade')::int = company.senior_id
WHERE
item_id = {book_id};

Specify FLOAT column precision in Peewee with MariaDB/MySQL

I am trying to specify the float precision for a column definition in Peewee and cannot find how to do this in the official docs or in the github issues.
My example model is below:
DB = peewee.MySQLDatabase(
"example",
host="localhost",
port=3306,
user="root",
password="whatever"
)
class TestModel(peewee.Model):
class Meta:
database = DB
value = peewee.FloatField()
The above creates the following table spec in the database:
SHOW COLUMNS FROM testmodel;
/*
+-------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+----------------+
| value | float | NO | | NULL | |
+-------+---------+------+-----+---------+----------------+
*/
What I would like is to specify the M and D parameters that the FLOAT field accepts so that the column is created with the precision parameters I need. I can accomplish this in SQL after the table is created using the below:
ALTER TABLE testmodel MODIFY COLUMN value FLOAT(20, 6); -- 20 and 6 are example parameters
Which gives this table spec:
SHOW COLUMNS FROM testmodel;
/*
+-------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+----------------+
| value | float(20,6) | YES | | NULL | |
+-------+-------------+------+-----+---------+----------------+
*/
But I'd like it be done at table creation time within the peewee structure itself, rather than needing to run a separate "alter table" query after the peewee.Database.create_tables() method is run. If there is no way to do this in the peewee.FloatField itself then I'd also accept any other solution so long as it ensures the create_tables() call will create the columns with the specified precision.

As #booshong already mentions
The simpelst solution is to subclass the default FloatField like this :
class CustomFloatField(FloatField):
def __init__(self, *args, **kwargs):
self.max_digits = kwargs.pop("max_digits", 7)
self.decimal_places = kwargs.pop("decimal_places", 4)
super().__init__(*args, **kwargs)
def get_modifiers(self):
return [self.max_digits, self.decimal_places]
and then use it like this
my_float_field = CustomFloatField(max_digits=2, decimal_places=2)

Django retrieve rows for the distinct column values

I want to query Model rows in Django,
class Language(models.Model):
language_id = models.CharField(max_length=100, default="")
code = models.CharField(max_length=100, default="")
name = models.CharField(max_length=500, default="")
In this table, the language_id is not unique, for example, below is the sample data
+-------------+------+---------+
| language_id | code | name |
+-------------+------+---------+
| 12345 | en | english |
| 12345 | te | telugu |
| 54321 | en | english |
| 54321 | te | telugu |
+-------------+------+---------+
I want to filter the rows(all columns) which should have distinct language_ids.
What currently I am doing.
language_list = Language.objects.all()
list = []
idlist = []
for language in language_list:
if language.language_id not in idlist:
il = language
list.append(il)
idlist.append(language.language_id)
Then list will have all the distinct rows(model objects).
Is there any better way to do this. I don't want to rotate through all the language models.

It's unclear what you are trying to do.
What your script does is take the first occurrence of a given ID arbitrarily.
If that's what you want, it will depend on what database your model is based.
PostgreSQL allows the use of distinct on a field:
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#distinct
On MySQL what you could do is get all the unique instances of your id and get an instance of your model matching once per ID:
language_ids = Language.objects.values_list('language_id', flat=True).distinct()
result = []
for language_id in language_ids:
result.append(Language.objects.filter(language_id=language_id).first())
It's not necessarily much better than your solution simply because arbitrary picking isn't an expected use case for the ORM.
If on the other hand you meant to only get language_ids that appear once and only once:
Language.objects.values('language_id').annotate(cnt=Count('id')).filter(cnt=1)

How to add DateTimeField in django without microsecond

I'm writing django application in django 1.8 and mysql 5.7.
Below is the model which I have written:
class People(models.Model):
name = models.CharField(max_length=20)
age = models.IntegerField()
create_time = models.DateTimeField()
class Meta:
db_table = "people"
Above model creates the table below:
mysql> desc people;
+-------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(20) | NO | | NULL | |
| age | int(11) | NO | | NULL | |
| create_time | datetime(6) | NO | | NULL | |
+-------------+-------------+------+-----+---------+----------------+
Here Django creates datetime field with microsecond
datetime(6)
But I want datetime field without microsecond
datetime
I have another application, which is also using the same database and that datetime field with microsecond is raising an issue for me.

This is really very interesting question. I looked through the source code and here is the reason for setting the datetime with fractional seconds. The following snippet is from the file django/db/backends/mysql/base.py:
class DatabaseWrapper(BaseDatabaseWrapper):
vendor = 'mysql'
# This dictionary maps Field objects to their associated MySQL column
# types, as strings. Column-type strings can contain format strings; they'll
# be interpolated against the values of Field.__dict__ before being output.
# If a column type is set to None, it won't be included in the output.
_data_types = {
'AutoField': 'integer AUTO_INCREMENT',
'BinaryField': 'longblob',
'BooleanField': 'bool',
'CharField': 'varchar(%(max_length)s)',
'CommaSeparatedIntegerField': 'varchar(%(max_length)s)',
'DateField': 'date',
'DateTimeField': 'datetime',
'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
'DurationField': 'bigint',
'FileField': 'varchar(%(max_length)s)',
'FilePathField': 'varchar(%(max_length)s)',
'FloatField': 'double precision',
'IntegerField': 'integer',
'BigIntegerField': 'bigint',
'IPAddressField': 'char(15)',
'GenericIPAddressField': 'char(39)',
'NullBooleanField': 'bool',
'OneToOneField': 'integer',
'PositiveIntegerField': 'integer UNSIGNED',
'PositiveSmallIntegerField': 'smallint UNSIGNED',
'SlugField': 'varchar(%(max_length)s)',
'SmallIntegerField': 'smallint',
'TextField': 'longtext',
'TimeField': 'time',
'UUIDField': 'char(32)',
}
#cached_property
def data_types(self):
if self.features.supports_microsecond_precision:
return dict(self._data_types, DateTimeField='datetime(6)', TimeField='time(6)')
else:
return self._data_types
# ... further class methods
In the method data_types the if condition checks the MySQL version. The method supports_microsecond_precision comes from the file django/db/backends/mysql/features.py:
class DatabaseFeatures(BaseDatabaseFeatures):
# ... properties and methods
def supports_microsecond_precision(self):
# See https://github.com/farcepest/MySQLdb1/issues/24 for the reason
# about requiring MySQLdb 1.2.5
return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
So when you use MySQL 5.6.4 or higher the field DateTimeField is mapped to datetime(6).
I couldn't find any possibility given by Django to adjust this, so ended up with monkey patching:
from django.db.backends.mysql.base import DatabaseWrapper
DatabaseWrapper.data_types = DatabaseWrapper._data_types
Put the above code where it suits best your needs, be it models.py or __init__.py, or maybe some other file.
When running migrations Django will create column datetime and not datetime(6) for DateTimeField, even if you're using MySQL 5.7.

This answer gave me an idea. What if you try to manually change the migrations.
First run python manage.py makemigrations and after that edit the file 0001_initial.py (or whatever the name is) in the subdirectory migrations of your app:
class Migration(migrations.Migration):
operations = [
migrations.CreateModel(
name = 'People'
fields = [
# the fields
# ... in this part comment or delete create_time
],
),
migrations.RunSQL(
"ALTER TABLE people ADD COLUMN create_time datetime(0)",
reverse_sql="ALTER TABLE people DROP COLUMN create_time",
state_operations=[
migrations.AddField(
model_name='people',
name='create_time',
fields= models.DateTimeField(),
)
]
)
]
This is just an example. You can try with different options and check with:
python manage.py sqlmigrations yourapp 0001
what the SQL output is. Instead of yourapp and 0001 provide the name of your app and the number of the migration.
Here is a link to the official documentation about fractional seconds time values.
EDIT: I tested the code above with MySQL 5.7 and it works as expected. Maybe it can help someone else. If you get some errors, check that you have installed mysqlclient and sqlparse.

SQLAlchemy relationships populating foreign key fields

I have the following tables with their respective sqlalchemy classes:
class Enrolled(Base):
__tablename__ = 'enrolled'
id = Column(Integer, primary_key=True, nullable=False, autoincrement=True)
student_fk = Column(Integer, ForeignKey('students.id'))
student = relationship('Students', foreign_keys=[device_fk], uselist=False,backref="enrolled", innerjoin=False, post_update=False)
subject = Column(String(5, convert_unicode=True), nullable=False)
//__init__ for id and subject is here.
class Students(Base):
__tablename__ = 'students'
id = Column(Integer, primary_key=True, nullable=False, autoincrement=True)
name = Column(String(50, convert_unicode=True), nullable=False)
//init for name is here
Relationship between students and enrolled is one to many. i.e one student can enroll himself in more then 1 subject.
Now, I know to insert a couple of subjects into 'Enrolled' and names into 'Students' classes.
DBSession.add(Enrolled(subject="maths"))
In the end this is how my tables look
Enrolled:
+----+------------+---------+
| id | student_fk | subject |
+----+------------+---------+
| 1 | | Maths |
| 2 | | Physics |
| 3 | | Art |
+----+------------+---------+
Students:
+----+------+
| id | name |
+----+------+
| 1 | Jim |
| 2 | Bob |
| 3 | Cara |
+----+------+
Now, how do I get the students id get into Enrolled table as foreign keys?
I have this information : which student is enrolled into which subject as a .csv file..
mycsv: name,subject,name1,subject1,name2,subject2
Should I have a manual dictionary like dict {jim:maths,Bob:Art,Cara:Physics} and then map like
query=Enrolled(subject="maths")
for k, v in dict.items():
if subject in v:
list.append(k)
for i in list:
query.student=DBSession.query(Students).filter(name=i).first()
DBSession.add(query)
Please help.. How do I get the student_fk field populated properly?

Your 1-to-many enrollment table should have composite primary key on Student ID and subject. Assuming you want to keep subjects as ENUM (which works with small list of subjects, otherwise you should move it to a separate table), you tables should look something like:
subjects = [ 'Maths', 'Physics', 'Art', ]
class Student(Base):
__tablename__ = 'Student'
student_id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String(50, convert_unicode=True), nullable=False)
class StudentEnrollment(Base):
__tablename__ = 'StudentEnrollment'
student_id = Column(Integer, ForeignKey('Student.student_id', ondelete='CASCADE'), primary_key=True)
subject = Column(Enum(*subjects), primary_key=True)
student = relationship("Student", primaryjoin='StudentEnrollment.student_id==Student.student_id', uselist=True, backref="enrollments")
which will result in:
root#localhost [inDB]> show create table Student\G
*************************** 1. row ***************************
Table: Student
Create Table: CREATE TABLE `Student` (
`student_id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(50) NOT NULL,
PRIMARY KEY (`student_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
root#localhost [inDB]> show create table StudentEnrollment\G
*************************** 1. row ***************************
Table: StudentEnrollment
Create Table: CREATE TABLE `StudentEnrollment` (
`student_id` int(11) NOT NULL,
`subject` enum('Maths','Physics','Art') NOT NULL,
PRIMARY KEY (`student_id`,`subject`),
CONSTRAINT `StudentEnrollment_ibfk_1` FOREIGN KEY (`student_id`) REFERENCES `Student` (`student_id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
then to insert few enrollments for user Jim:
student = Student(name='Jim')
session.add(student)
session.flush()
for enr in ('Maths', 'Physics', 'Art'):
session.add(StudentEnrollment(student_id=student.student_id, subject=enr))
session.flush()
session.commit()
which will result in:
root#localhost [inDB]> select * from Student;
+------------+------+
| student_id | name |
+------------+------+
| 3 | Jim |
+------------+------+
1 row in set (0.00 sec)
root#localhost [inDB]> select * from StudentEnrollment;
+------------+---------+
| student_id | subject |
+------------+---------+
| 3 | Maths |
| 3 | Physics |
| 3 | Art |
+------------+---------+
3 rows in set (0.00 sec)
This is a very basic example with two tables. A better option would be to normalize Enrollments into separate table and use association proxy pattern, see http://docs.sqlalchemy.org/en/rel_0_9/orm/extensions/associationproxy.html