I'm writing django application in django 1.8 and mysql 5.7.
Below is the model which I have written:
class People(models.Model):
name = models.CharField(max_length=20)
age = models.IntegerField()
create_time = models.DateTimeField()
class Meta:
db_table = "people"
Above model creates the table below:
mysql> desc people;
+-------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(20) | NO | | NULL | |
| age | int(11) | NO | | NULL | |
| create_time | datetime(6) | NO | | NULL | |
+-------------+-------------+------+-----+---------+----------------+
Here Django creates datetime field with microsecond
datetime(6)
But I want datetime field without microsecond
datetime
I have another application, which is also using the same database and that datetime field with microsecond is raising an issue for me.
This is really very interesting question. I looked through the source code and here is the reason for setting the datetime with fractional seconds. The following snippet is from the file django/db/backends/mysql/base.py:
class DatabaseWrapper(BaseDatabaseWrapper):
vendor = 'mysql'
# This dictionary maps Field objects to their associated MySQL column
# types, as strings. Column-type strings can contain format strings; they'll
# be interpolated against the values of Field.__dict__ before being output.
# If a column type is set to None, it won't be included in the output.
_data_types = {
'AutoField': 'integer AUTO_INCREMENT',
'BinaryField': 'longblob',
'BooleanField': 'bool',
'CharField': 'varchar(%(max_length)s)',
'CommaSeparatedIntegerField': 'varchar(%(max_length)s)',
'DateField': 'date',
'DateTimeField': 'datetime',
'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
'DurationField': 'bigint',
'FileField': 'varchar(%(max_length)s)',
'FilePathField': 'varchar(%(max_length)s)',
'FloatField': 'double precision',
'IntegerField': 'integer',
'BigIntegerField': 'bigint',
'IPAddressField': 'char(15)',
'GenericIPAddressField': 'char(39)',
'NullBooleanField': 'bool',
'OneToOneField': 'integer',
'PositiveIntegerField': 'integer UNSIGNED',
'PositiveSmallIntegerField': 'smallint UNSIGNED',
'SlugField': 'varchar(%(max_length)s)',
'SmallIntegerField': 'smallint',
'TextField': 'longtext',
'TimeField': 'time',
'UUIDField': 'char(32)',
}
#cached_property
def data_types(self):
if self.features.supports_microsecond_precision:
return dict(self._data_types, DateTimeField='datetime(6)', TimeField='time(6)')
else:
return self._data_types
# ... further class methods
In the method data_types the if condition checks the MySQL version. The method supports_microsecond_precision comes from the file django/db/backends/mysql/features.py:
class DatabaseFeatures(BaseDatabaseFeatures):
# ... properties and methods
def supports_microsecond_precision(self):
# See https://github.com/farcepest/MySQLdb1/issues/24 for the reason
# about requiring MySQLdb 1.2.5
return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
So when you use MySQL 5.6.4 or higher the field DateTimeField is mapped to datetime(6).
I couldn't find any possibility given by Django to adjust this, so ended up with monkey patching:
from django.db.backends.mysql.base import DatabaseWrapper
DatabaseWrapper.data_types = DatabaseWrapper._data_types
Put the above code where it suits best your needs, be it models.py or __init__.py, or maybe some other file.
When running migrations Django will create column datetime and not datetime(6) for DateTimeField, even if you're using MySQL 5.7.
This answer gave me an idea. What if you try to manually change the migrations.
First run python manage.py makemigrations and after that edit the file 0001_initial.py (or whatever the name is) in the subdirectory migrations of your app:
class Migration(migrations.Migration):
operations = [
migrations.CreateModel(
name = 'People'
fields = [
# the fields
# ... in this part comment or delete create_time
],
),
migrations.RunSQL(
"ALTER TABLE people ADD COLUMN create_time datetime(0)",
reverse_sql="ALTER TABLE people DROP COLUMN create_time",
state_operations=[
migrations.AddField(
model_name='people',
name='create_time',
fields= models.DateTimeField(),
)
]
)
]
This is just an example. You can try with different options and check with:
python manage.py sqlmigrations yourapp 0001
what the SQL output is. Instead of yourapp and 0001 provide the name of your app and the number of the migration.
Here is a link to the official documentation about fractional seconds time values.
EDIT: I tested the code above with MySQL 5.7 and it works as expected. Maybe it can help someone else. If you get some errors, check that you have installed mysqlclient and sqlparse.
Related
I am trying to specify the float precision for a column definition in Peewee and cannot find how to do this in the official docs or in the github issues.
My example model is below:
DB = peewee.MySQLDatabase(
"example",
host="localhost",
port=3306,
user="root",
password="whatever"
)
class TestModel(peewee.Model):
class Meta:
database = DB
value = peewee.FloatField()
The above creates the following table spec in the database:
SHOW COLUMNS FROM testmodel;
/*
+-------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+----------------+
| value | float | NO | | NULL | |
+-------+---------+------+-----+---------+----------------+
*/
What I would like is to specify the M and D parameters that the FLOAT field accepts so that the column is created with the precision parameters I need. I can accomplish this in SQL after the table is created using the below:
ALTER TABLE testmodel MODIFY COLUMN value FLOAT(20, 6); -- 20 and 6 are example parameters
Which gives this table spec:
SHOW COLUMNS FROM testmodel;
/*
+-------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+----------------+
| value | float(20,6) | YES | | NULL | |
+-------+-------------+------+-----+---------+----------------+
*/
But I'd like it be done at table creation time within the peewee structure itself, rather than needing to run a separate "alter table" query after the peewee.Database.create_tables() method is run. If there is no way to do this in the peewee.FloatField itself then I'd also accept any other solution so long as it ensures the create_tables() call will create the columns with the specified precision.
As #booshong already mentions
The simpelst solution is to subclass the default FloatField like this :
class CustomFloatField(FloatField):
def __init__(self, *args, **kwargs):
self.max_digits = kwargs.pop("max_digits", 7)
self.decimal_places = kwargs.pop("decimal_places", 4)
super().__init__(*args, **kwargs)
def get_modifiers(self):
return [self.max_digits, self.decimal_places]
and then use it like this
my_float_field = CustomFloatField(max_digits=2, decimal_places=2)
On the database i have 3 tables:
languages
cities
city_language
city_language Table:
+-------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+-------+
| city_id | bigint(20) unsigned | NO | PRI | NULL | |
| language_id | bigint(20) unsigned | NO | PRI | NULL | |
| name | varchar(255) | NO | | NULL | |
+-------------+---------------------+------+-----+---------+-------+
Model
class CityLanguage(models.Model):
city = models.ForeignKey('Cities', models.DO_NOTHING)
language = models.ForeignKey('Languages', models.DO_NOTHING)
name = models.CharField(max_length=255)
class Meta:
managed = False
db_table = 'city_language'
unique_together = (('city', 'language'),)
Model doesn't have id field and primary key also my table doesn't have id column. If i run this code i got error:
(1054, "Unknown column 'city_language.id' in 'field list'")
If i define primary key for a column this column values should unique. If i use primary_key when i want to put same city with different languages i get
With this city (name or language it depends on which column choose for primary key) already exists.
I don't want to create id column for pivot table. There is no reason create id column for pivot table. Please can you tell me how can i use pivot table with correct way. Thank you.
Django without primary_key not work. There is two way to figure out it:
Create id (Then Django model you don't need to add primary key)
Create other unique column and set it primary key, and also made it unique.
On my side i choose second way created a column named: unique_key and in model put the code.
unique_key = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
you need to import uuid.
Good luck.
I have the following django data model
class ApiLog(models.Model):
name = models.TextField(blank=False, null=False)
ts = models.DateTimeField(default=timezone.now, blank=False, null=False)
ip_address = models.GenericIPAddressField(blank=True, null=True)
user = models.ForeignKey(User, on_delete=models.CASCADE)
It ends up with database
django=# \d+ users_apilog
Table "public.users_apilog"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------------------+----------+--------------+-------------
id | integer | | not null | nextval('users_apilog_id_seq'::regclass) | plain | |
name | text | | not null | | extended | |
ts | timestamp with time zone | | not null | | plain | |
ip_address | inet | | | | main | |
user_id | integer | | not null | | plain | |
Indexes:
"users_apilog_pkey" PRIMARY KEY, btree (id)
"users_apilog_user_id_2eb2b1cf" btree (user_id)
Foreign-key constraints:
"users_apilog_user_id_2eb2b1cf_fk_users_user_id" FOREIGN KEY (user_id) REFERENCES users_user(id) DEFERRABLE INITIALLY DEFERRED
Since, this table will also be accessed by non-django app. I need to make sure the auto timestamp generation (For column ts) is fully handled by postgres, not python.
I don't expect to have
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
------------+--------------------------+-----------+----------+---------+---------+--------------+-------------
ts | timestamp with time zone | | not null | | plain | |
I expect to have
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
------------+--------------------------+-----------+----------+---------+---------+--------------+-------------
ts | timestamp with time zone | | not null | now() | plain | |
I had tried other technique like auto_now=True, auto_now_add=True, ...
But, none of them generate the table schema I want. By using any below, the generated table schema, its default column is still empty.
default=timezone.now
auto_now=True
auto_now_add=True
Since, it is pretty common to use docker these day. I demonstrate how I solve it, based on docker development environment.
Step 0 : Make sure you had already committed generated 0001_initial.py
Step 1 : Generate empty migration file
docker-compose run --rm -v %cd%/django:/app -w /app django /app/manage.py makemigrations users --empty -n alter_ts_default_to_now
Step 2 : Use raw SQL in migration file
# Generated by Django 2.1 on 2018-08-25 09:27
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('users', '0001_initial'),
]
operations = [
migrations.RunSQL(
"ALTER TABLE users_apilog ALTER COLUMN ts SET DEFAULT now()",
)
]
Step 3 : Run migration again
#!/bin/sh
python manage.py makemigrations users
python manage.py makemigrations
python manage.py migrate
echo "yes" | python manage.py collectstatic
exec "$#"
I have code like below.
from uuid import uuid4
from uuid import uuid1
from cassandra.cqlengine import columns, connection
from cassandra.cqlengine.models import Model
from cassandra.cqlengine.management import sync_table
class BaseModel(Model):
__abstract__ = True
id = columns.UUID(primary_key=True, default=uuid4)
created_timestamp = columns.TimeUUID(primary_key=True,
clustering_order='DESC',
default=uuid1)
deleted = columns.Boolean(required=True, default=False)
class OtherModel(BaseModel):
__table_name__ = 'other_table'
if __name__ == '__main__':
connection.setup(hosts=['localhost'],
default_keyspace='test')
sync_table(OtherModel)
OtherModel.create()
After first execution, I can see the record in db when run query as.
cqlsh> select * from test.other_table;
id | created_timestamp | deleted
--------------------------------------+--------------------------------------+---------
febc7789-5806-44d8-bbd5-45321676def9 | 840e1b66-cc73-11e6-a66c-38c986054a88 | False
(1 rows)
After this, I added new column name in OtherModel it and run same program.
class OtherModel(BaseModel):
__table_name__ = 'other_table'
name = columns.Text(required=True, default='')
if __name__ == '__main__':
connection.setup(hosts=['localhost'],
default_keyspace='test')
sync_table(OtherModel)
OtherModel.create(name='test')
When check db entry
cqlsh> select * from test.other_table;
id | created_timestamp | deleted | name
--------------------------------------+--------------------------------------+---------+------
936cfd6c-44a4-43d3-a3c0-fdd493144f4b | 4d7fd78c-cc74-11e6-bb49-38c986054a88 | False | test
febc7789-5806-44d8-bbd5-45321676def9 | 840e1b66-cc73-11e6-a66c-38c986054a88 | False | null
(2 rows)
There is one row with name as null.
But I can't query on null value.
cqlsh> select * from test.other_table where name=null;
InvalidRequest: code=2200 [Invalid query] message="Unsupported null value for indexed column name"
I got reference How Can I Search for Records That Have A Null/Empty Field Using CQL?.
When I set default='' in the Model, why it not set for all the null value in table?
Is there any way to set null value in name to default value '' with query?
The null cell is actually it just not being set. And the absence of data isn't something you can query on, since its a filtering operation. Its not scalable or possible to do efficiently, so its not something C* will encourage (or in this case even allow).
Going back and retroactively setting values to all the previously created rows would be very expensive (has to read everything, then do as many writes). Its pretty easy in application side to just say if name is null its '' though.
I'm working on a small pet-project that involves some accounting in multiple currencies. During its development I decided to move from straight-forward DB setting to DB-migrations using alembic. And on some migrations I need to populate DB with initial currencies, that are displayed in Ukrainian.
My problem is that data populated from alembic migration scripts is saving in some unknown encoding, so I cannot use it within the application (that expects to be human readable). My settings as well as script are as follows:
alembic.ini
...
sqlalchemy.url = mysql+pymysql://defaultuser:defaultpwd#localhost/petdb
...
alembic/versions/f433ab2a814_adding_currency.py
from alembic import op
# -*- coding: utf-8 -*-
"""Adding currency
Revision ID: f433ab2a814
Revises: 49538bba2220
Create Date: 2016-03-08 13:50:35.369021
"""
# revision identifiers, used by Alembic.
revision = 'f433ab2a814'
down_revision = '1c0b47263c82'
branch_labels = None
depends_on = None
def upgrade():
op.create_table(
'currency',
Column('id', Integer, primary_key=True),
Column('name', Unicode(120), nullable=False),
Column('abbr', String(3), nullable=False)
)
op.execute(u'INSERT INTO currency SET name="{}", abbr="{}";'.format(u"Гривня", "UAH"))
After checking table currency from mysql client or mysql-workbench, it is displayed as:
mysql> SELECT * FROM currency;
+----+----------------------------+------+
| id | name | abbr |
+----+----------------------------+------+
| 1 | Ð“Ñ€Ð¸Ð²Ð½Ñ | UAH |
+----+----------------------------+------+
Expected result is:
mysql> SELECT * FROM currency;
+----+----------------------------+------+
| id | name | abbr |
+----+----------------------------+------+
| 1 | Гривня | UAH |
+----+----------------------------+------+
From my application I've been setting this value as follows:
from petproject import app
app.config.from_object(config.DevelopmentConfig)
engine = create_engine(app.config["DATABASE"]+"?charset=utf8",
convert_unicode=True, encoding="utf8", echo=False)
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
if len(db_session.query(Currency).all()) == 0:
default_currency = Currency()
default_currency.name = u"Гривня"
default_currency.abbr = u"UAH"
db_session.add(default_currency)
db_session.commit()
So I'm wondering how to insert initial Unicode values on migration that will be stored in correct encoding. Did I miss anything?
After a more extended analysis, I discovered, that MySQL keeps all data in 'windows-1252' encoding. MySQL manual (section "West European Character Sets") states about this issue as:
latin1 is the default character set. MySQL's latin1 is the same as the Windows cp1252 character set.
It looked like either MySQL ignored character_set_client that, I assumed to be 'utf-8', or SQLAlchemy / alembic didn't inform server to accept data as 'UTF-8' encoded data. Unfortunatelly, recommended option '?charset=utf8' is not possible to set in alembic.ini.
In order to accept and save data in correct encoding, I set character set manually by calling op.execute('SET NAMES utf8');. Thus complete code looks like:
def upgrade():
op.create_table(
'currency',
Column('id', Integer, primary_key=True),
Column('name', Unicode(120), nullable=False),
Column('abbr', String(3), nullable=False)
)
op.execute('SET NAMES utf8')
op.execute(u'INSERT INTO currency SET name="{}", abbr="{}";'.format(u"Гривня", "UAH"))
And result became as expected:
mysql> SELECT * FROM currency;
+----+----------------------------+------+
| id | name | abbr |
+----+----------------------------+------+
| 1 | Гривня | UAH |
+----+----------------------------+------+