I use a simple MySQL Database with a SQLAlchemy Model:
from tokenize import String
from sqlalchemy import Column, Integer, String
from .database import Base
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True, index=True)
email = Column(String(256), unique=True, index=True)
hashed_password = Column(String(256))
It works fine, but when the character length exceeds a number of characters which is much smaller than 256, I get the following error:
fastapi | sqlalchemy.exc.DataError: (MySQLdb._exceptions.DataError) (1406, "Data too long for column 'email' at row 1")
fastapi | [SQL: INSERT INTO users (email, hashed_password) VALUES (%s, %s)]
fastapi | [parameters: ('sdasdasdasdasdasda', 'sdasdasdasdasdasdsadadasdasd')]
I know there is a "strict-mode" in MySQL, but I would rather like to understand the error here and make it work in the python code, since I run the Database in a Container.
SHOW CREATE TABLE users:
| users | CREATE TABLE `users` (
`id` int NOT NULL AUTO_INCREMENT,
`email` varchar(256) DEFAULT NULL,
`hashed_password` varchar(256) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ix_users_email` (`email`),
KEY `ix_users_id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
Related
total newbie to Alembic, SQLAlchemy, and Python. I've gotten to the point where Alembic is comparing existing objects in the database against the declarative classes I've made, and there's one pesky index (for a foreign key) that Alembic refuses to leave in-place in my initial migration.
I'm completely at a loss as to why the migration is continually trying to drop and re-create this index, which, if I leave in the migration I'll wager is going to fail anyway. Plus, if I don't reconcile the class to the database this will likely come up every time I auto-generate migrations.
Here's the pertinent part of what is in the upgrade method:
op.drop_index(
'vndr_prod_tp_cat_category_fk_idx',
table_name='vendor_product_types_magento_categories'
)
In the downgrade method:
op.create_index(
'vndr_prod_tp_cat_category_fk_idx',
'vendor_product_types_magento_categories',
['magento_category_id'],
unique=False
)
...here's the DDL for the table as it exists in MySQL:
CREATE TABLE `vendor_product_types_magento_categories` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`vendor_product_type_id` bigint(20) unsigned NOT NULL,
`magento_category_id` bigint(20) unsigned NOT NULL,
`sequence` tinyint(3) unsigned NOT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `vendor_product_types_magento_categories_uq` (`vendor_product_type_id`,`magento_category_id`,`sequence`),
KEY `vndr_prod_tp_cat_category_fk_idx` (`magento_category_id`),
CONSTRAINT `vndr_prod_tp_cat_magento_category_fk` FOREIGN KEY (`magento_category_id`) REFERENCES `magento_categories` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `vndr_prod_tp_cat_product_type_fk` FOREIGN KEY (`vendor_product_type_id`) REFERENCES `vendor_product_types` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=101 DEFAULT CHARSET=utf8
...and here's the class I wrote:
from sqlalchemy import Column, Integer, UniqueConstraint, ForeignKeyConstraint, Index
from sqlalchemy.dialects.mysql import TIMESTAMP
from sqlalchemy.sql import text
from .base import Base
class VendorProductTypesMagentoCategories(Base):
__tablename__ = 'vendor_product_types_magento_categories'
id = Column(Integer, primary_key=True)
vendor_product_type_id = Column(
Integer,
nullable=False
)
magento_category_id = Column(
Integer,
nullable=False
)
sequence = Column(Integer, nullable=False)
created_at = Column(TIMESTAMP, server_default=text('CURRENT_TIMESTAMP'), nullable=False)
updated_at = Column(
TIMESTAMP,
server_default=text('NULL ON UPDATE CURRENT_TIMESTAMP'),
nullable=True
)
__table_args__ = (
UniqueConstraint(
'vendor_product_type_id',
'magento_category_id',
'sequence',
name='vendor_product_types_magento_categories_uq'
),
ForeignKeyConstraint(
('vendor_product_type_id',),
('vendor_product_types.id',),
name='vndr_prod_tp_cat_product_type_fk'
),
ForeignKeyConstraint(
('magento_category_id',),
('magento_categories.id',),
name='vndr_prod_tp_cat_category_fk_idx'
),
)
def __repr__(self):
return '<VendorProductTypesMagentoCategories (id={}, vendor_name={}, product_type={})>'.format(
self.id,
self.vendor_name,
self.product_type
)
You define your product foreign key in your python code as
ForeignKeyConstraint(
('magento_category_id',),
('magento_categories.id',),
name='vndr_prod_tp_cat_category_fk_idx'
)
Here you use vndr_prod_tp_cat_category_fk_idx as the name of the foreign key constraint, not as the name of the underlying index, which explains why sqlalchemy wants to drop the index.
You should use vndr_prod_tp_cat_product_type_fk as the foreign key name and have a separate Index() construct with vndr_prod_tp_cat_category_fk_idx as name to create the index.
I am building a flask application over an already existing database so there was no need to declare the whole models fully. I have this table:
class Users(db.Model):
__tablename__ = 'my_users'
__table_args__ = {
'autoload': True,
'autoload_with': db.engine
}
the table has about 10 columns but when i query the data i can see that the attribute:
.__dict__
only returns the first 4 columns. i have tried using filter and also filter by but data returned only contains the first 4 columns. Here is my query:
users = Users.query.filter(
section_serial == sectionserial
).all()
I am using the postgres database. Here is a minimal example:
CREATE TABLE public.my_users
(
user_serial integer NOT NULL DEFAULT nextval('my_users_seq'::regclass),
user_name character varying(16) NOT NULL,
user_password character varying(42) NOT NULL,
id_number character varying(155) NOT NULL,
date_added timestamp without time zone NOT NULL DEFAULT now(),
is_enabled boolean NOT NULL DEFAULT true,
expiry_date date NOT NULL DEFAULT (('now'::text)::date + 30),
phone_number character varying(254),
notes text,
section_serial integer,
full_name character varying(155) NOT NULL,
zip_code boolean NOT NULL DEFAULT false,
CONSTRAINT user_serial_pkey PRIMARY KEY (user_serial)
);
After querying the data i only get user_serial, user_name, user_password and id_number. I cannot get the rest of the columns
The problem was it was conflicting with a login model i had created though with a different name. I think models should just be declared once.
class SystemUsers(db.Model):
__tablename__ = 'my_users'
userserial = db.Column(
'user_serial', db.Integer, primary_key=True)
username = db.Column('user_name ', db.String)
password= db.Column('user_password ', db.String)
idnumber = db.Column('id_number', db.String)
isactive = True
isanonymous = False
authenticated = False
I cannot get my very basic SQL query to work as it returns 0 values despite the fact that there are clearly nulls
query
SELECT
*
FROM
leads AS l
JOIN closes c ON l.id = c.lead_id
WHERE
c.close_date IS NULL
DDL
CREATE TABLE closes
(
id INT AUTO_INCREMENT
PRIMARY KEY,
lead_id INT NOT NULL,
close_date DATETIME NULL,
close_type VARCHAR(255) NULL,
primary_agent VARCHAR(255) NULL,
price FLOAT NULL,
gross_commission FLOAT NULL,
company_dollar FLOAT NULL,
address VARCHAR(255) NULL,
city VARCHAR(255) NULL,
state VARCHAR(10) NULL,
zip VARCHAR(10) NULL,
CONSTRAINT closes_ibfk_1
FOREIGN KEY (lead_id) REFERENCES leads (id)
)
ENGINE = InnoDB;
CREATE INDEX lead_id
ON closes (lead_id);
I should mention that I am inserting the data with a python web scraper and SQLAlchemy. If the data is not scraped it will be None on insert.
Here is a screenshot of datagrip showing a null value in the row
EDIT
Alright so I went ahead and ran the following on some of the entries in the table where the value was already <null>
UPDATE closes
SET close_date = NULL
WHERE
lead_id = <INTEGERVAL>
;
What is interesting now is that when running the original query I do actually return the 2 records that I ran the update query for (the expected outcome). This would lead me to believe that the issues is with how my SQLAlchemy model is mapping the values on insert.
models.py
class Close(db.Model, ItemMixin):
__tablename__ = 'closes'
id = db.Column(db.Integer, primary_key=True)
lead_id = db.Column(db.Integer, db.ForeignKey('leads.id'), nullable=False)
close_date = db.Column(db.DateTime)
close_type = db.Column(db.String(255))
primary_agent = db.Column(db.String(255))
price = db.Column(db.Float)
gross_commission = db.Column(db.Float)
company_dollar = db.Column(db.Float)
address = db.Column(db.String(255))
city = db.Column(db.String(255))
state = db.Column(db.String(10))
zip = db.Column(db.String(10))
def __init__(self, item):
self.build_from_item(item)
def build_from_item(self, item):
for k, v in item.items():
setattr(self, k, v)
But I am fairly confident the value is a python None in the event no value is scraped from the website. My understanding is that SQLAlchemy would map a None to NULL on insert and given that nullable=True is the default setting, which can seen on the generated DDL, I am still at a loss as to why it appears to be NULL when in reality it is not behaving that way.
EDIT 2
Only place where I think the issue would be happening is where my spider actually scrapes the data and assigns it to the Item which is shown below
closes.py
# item['close_date'] = None at this point
try:
item['close_date'] = arrow.get(item['close_date'], 'MMM D, YYYY').format('YYYY-MM-DD')
except ParserError as e:
# Maybe item['close_date'] = None here?
spider.logger.error(f'Parse error: {item["close_date"]} - {e}')
In the python code I've written this would appear to be the place where the issue would arise. But if arrow.get throws an exception the value of item['close_date'] should still be None. If that is not the case and even if it is it does not explain why it appears that the record value is NULL even thought it does not behave like it is.
I'm guessing that you're having an issue with the join, not the NULL value. The query below returns 1 result for me. More info about your data, the software used for querying (I tested with SQL Yog), and applicable versions might help.
EDIT
It could be that you're having issues with MySQL's 'zero date'.
https://dev.mysql.com/doc/refman/5.7/en/date-and-time-types.html
MySQL permits you to store a “zero” value of '0000-00-00' as a “dummy
date.” This is in some cases more convenient than using NULL values,
and uses less data and index space. To disallow '0000-00-00', enable
the NO_ZERO_DATE mode.
I've updated the SQL data below to include a zero date in the INSERT and SELECT's WHERE.
DROP TABLE IF EXISTS closes;
DROP TABLE IF EXISTS leads;
CREATE TABLE leads (
id INT(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (id)
) ENGINE=INNODB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8;
INSERT INTO leads(id) VALUES (1),(2),(3);
CREATE TABLE closes (
id INT(11) NOT NULL AUTO_INCREMENT,
lead_id INT(11) NOT NULL,
close_date DATETIME DEFAULT NULL,
close_type VARCHAR(255) DEFAULT NULL,
primary_agent VARCHAR(255) DEFAULT NULL,
price FLOAT DEFAULT NULL,
gross_commission FLOAT DEFAULT NULL,
company_dollar FLOAT DEFAULT NULL,
address VARCHAR(255) DEFAULT NULL,
city VARCHAR(255) DEFAULT NULL,
state VARCHAR(10) DEFAULT NULL,
zip VARCHAR(10) DEFAULT NULL,
PRIMARY KEY (id),
KEY lead_id (lead_id),
CONSTRAINT closes_ibfk_1 FOREIGN KEY (lead_id) REFERENCES leads (id)
) ENGINE=INNODB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1;
INSERT INTO closes(id,lead_id,close_date,close_type,primary_agent,price,gross_commission,company_dollar,address,city,state,zip)
VALUES
(1,3,'0000-00-0000',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL),
(2,1,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL),
(3,2,'2018-01-09 17:01:44',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
SELECT
*
FROM
leads AS l
JOIN closes c ON l.id = c.lead_id
WHERE
c.close_date IS NULL OR c.close_date = '0000-00-00';
I Try to save a hashed Password in postgresql database using SQL Alchemy.
table script is:
Create Table "User"(
Id serial Primary key,
UserName varchar(50) unique not null,
Nikname varchar(50) not null,
"password" varchar(172) not null,
FirstName varchar(75) not null,
LastName varchar(75) not null,
BirthDate date not null,
CreateDate date not null,
Status smallint Not null
)
and this is the mapping:
user = Table('User', metadata,
Column('id', Sequence(name='User_id_seq'), primary_key=True),
Column('username', String(50), unique=True, nullable=False),
Column('nikname', String(50), nullable=False),
Column('firstname', String(75), nullable=False),
Column('lastname', String(75), nullable=False),
Column('password', String(172), nullable=False),
Column('status', Integer, nullable=False),
Column('birthdate', Date, nullable=False),
Column('createdate', Date, nullable=False)
)
when i try to insert data, this exception raised :
sqlalchemy.exc.DataError: (psycopg2.DataError) value too long for type character varying(172)
[SQL: 'INSERT INTO "User" (id, username, nikname, firstname, lastname, password, status, birthdate, createdate) VALUES (nextval(\'"User_id_seq"\'), %(username)s, %(nikname)s, %(firstname)s, %(lastname)s, %(password)s, %(status)s, %(birthdate)s, %(createdate)s) RETURNING "User".id'] [parameters: {'username': 'hoseinyeganloo#gmail.com', 'nikname': 'Laughing Death', 'firstname': 'Hosein', 'lastname': 'Yegnloo', 'password': b'i1SlFeDkCZ0BJYanhINGCZC80rqVYABHAS/Ot2AWDgzPZCtshMNRZGHeosx3PvLqsCWzZfPZpsT+UZZLShmQxfbO5VJ4xJbLNjbb0n8HuazQy+0u5Ws2DCtmdDh+HFBTKCAiNuzUGueurP9d2VE3tHwHpX+hCMS1RB4KIOUORKw=', 'status': 1, 'birthdate': datetime.datetime(1990, 3, 1, 0, 0), 'createdate': datetime.datetime(2017, 6, 23, 0, 0)}]
but as you see data is exactly fit too field and there is no error when i execute this query inside pgadmin!
I think problem is in my mapping. i Changed String to Text but error resists :|
Any idea?
I don't know if it help. when all characters are digits, code work's with no error.
I try to insert some digits instead of hashed password and it works!
Update
I found that problem is character encoding! Some how SQLAlchemy increase size of passed string! Now i'am trying to prevent it!
Problem is not about mapping or charset or any things like that in sql Alchemy!
it is my code! when i try to convert hashing result to base64 string, result will be a BinaryString! not a String.
'password': b'i1SlFeDkCZ0BJYanhING....
So to solve this problem i need to decode base64 result to unicode string befor save it to database!
u.password = u.password.decode("utf-8", "ignore")
This issue arises because of postgresql and other ORM data structure definition which is strictly type and not loosely like SQL. I have identified the problems in two fold
db.Column definition
Posting Data by method should follow the model method args order
Where as the above may not affect SQLAlchemy db interactions migrating to other ORM requires you conform to the above.
Table Definition
__tablename__ = 'test'
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
accountNumber = db.Column(db.String(255))
description = db.Column(db.String(255), nullable=False)
accountType = db.Column(db.String(255), nullable=False)
Posting data by column role order
createAccount(accountNumber, description, accountType)
string supports 255 characters, while text supports 30000 characters - string vs text. I changed data type to db.Text but I still got the same error, so I dropped the database and created it again. Then run flask db init flask db migrate flask db upgrade
I am using sqlalchemy 0.7 and MySQL server version 5.1.63.
I have the following table on my database:
CREATE TABLE `g_domains` (
`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(255) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `name` (`name`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
The corresponding model is :
class GDomain(Base):
__tablename__ = 'g_domains'
__table_args__ = {
'mysql_engine': 'InnoDB',
'mysql_charset': 'utf8',
'mysql_collate': 'utf8_general_ci'
}
id = Column(mysql.BIGINT(unsigned=True), primary_key=True)
name = Column(mysql.VARCHAR(255, collation='utf8_general_ci'),
nullable=False, unique=True)
The following query in sql alchemy returns no rows :
session.query(GDomain).filter(GDomain.name.in_(domain_set)).
limit(len(domain_set)).all()
where domain_set is a python list containing some domain names like
domain_set = ['www.google.com', 'www.yahoo.com', 'www.AMAZON.com']
Although the table has a row (1, www.amazon.com) the above query returns only
(www.google.com, www.yahoo.com).
When I run the sql query :
SELECT * FROM g_domains
WHERE name IN ('www.google.com', 'www.yahoo.com', 'www.AMAZON.com')
Do you have an idea why this is happening?
Thanks in advance
What is the model_domain variable? Usually it looks like this:
session.query(GDomain).filter(GDomain.name.in_(domain_set)).
limit(len(domain_set)).all()
Note that the GDomain is used in both places. Alternatively you can use aliases:
domains = orm.aliased(GDomain, name='domain')
session.query(domains).filter(domains.name.in_(domain_set))
You can always try debugging, print the query that produced by sqlalchemy (see: SQLAlchemy: print the actual query)