I have 0 experience with postgresql and am deploying an app written in python using sqlalchemy to a server with postgres.
For development, I used an sqlite server.
Things are going pretty smoothly, but I hit a bump I don't know how to resolve.
I have three tables that look like that
class Car(db.Model):
id= db.Column(db.Integer, primary_key=True)
...
class Truck(db.Model):
id= db.Column(db.String(32), primary_key=True)
...
class Vehicles(db.Model):
id= db.Column(db.Integer, primary_key=True)
type= db.Column(db.String) #This is either 'car' or 'truck'
value= db.Column(db.String) #That's the car or truck id
...
I have a query that selects from Vehicles where type = 'car' AND value = 10
This is throwing an error:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) operator does not exist: integer = character varying
So I guess this is because Car.id is an int and Vehicle.value is a string..
How to write this query in sqlalchemy? Is there a way to write it and make it compatible with my sqlite dev environment and the pgsql production?
currently it looks like that
db.session.query(Vehicle).filter(Car.id == Vehicle.value)
PS: The truck id has to be a string and the car id has to be an int. I don't have control over that.
Simply cast to a string:
db.session.query(Vehicle).filter(str(Car.id) == Vehicle.value)
if Car.id is a local variable that is an int.
If you need to use this in a join, have the database cast it to a string:
from sqlalchemy.sql.expression import cast
db.session.query(Vehicle).filter(cast(Car.id, sqlalchemy.String) == Vehicle.value)
If the string value in the other column contains digits and possibly whitespace you may have to consider trimming, or instead casting the string value to an integer (and leave the integer column an integer).
Related
I'm using SQLAlchemy (Version: 1.4.44) and I'm having some unexpected results when trying to select columns and using cast on those columns.
First, most of the examples and even current documentation suggests column selection should work by passing an array to the select function like this:
s = select([table.c.col1])
However, I get the following error if I try this:
s = my_table.select([my_table.columns.user_id])
sqlalchemy.exc.ArgumentError: SQL expression for WHERE/HAVING role expected, got [Column('user_id', String(), table=<my_table>)].
Some examples suggest just placing the field directly in the select query.
s = select(table.c.col1)
But this seems to do nothing more than create an idle where-clause out of the field.
I eventually was able to achieve column selection with this approach:
s = my_table.select().with_only_columns(my_table.columns.created_at)
But I am not able to use cast for some reason with this approach.
s = my_table.select().with_only_columns(cast(my_table.columns.created_at, Date))
ValueError: Couldn't parse date string '2022' - value is not a string.
All help appreciated!
I don't think table.select() is common usage. SQLAlchemy is in a big transition right now on its way to 2.0. In 1.4 (and in 2) the following syntax should work, use whatever session handling you already have working I just mean the select(...):
from sqlalchemy.sql import select, cast
from sqlalchemy.dialects.postgresql import INTEGER
class User(Base):
__tablename__ = "users"
id = Column(
Integer, nullable=False, primary_key=True
)
name = Column(Text)
with Session(engine) as session:
u1 = User(name="1")
session.add(u1)
session.commit()
with Session(engine) as session:
my_table = User.__table__
# Cast user name into integer.
print (session.execute(select(cast(my_table.c.name, INTEGER))).all())
When I create an SQLite database from a python data model, any column defined as a String in Python is displayed as VARCHAR in SQLite (viewing with DB Browser for SQLite). Here is an example of the data model in Python:
class Users(db.Model):
id = db.Column(db.Integer, primary_key=True)
role = db.Column(db.String(10))
name_first = db.Column(db.String(50), nullable=False)
name_last = db.Column(db.String(50), nullable=False)
This may not be relevant, but I should clarify that I'm doing this as part of a website hosted with Flask. The database is initially created by dropping to a python prompt and:
from app import db
db.create_all()
I have a basic understanding of MS SQL and SQLite datatypes (NULL,INTEGER,REAL,TEXT,BLOB), but I don't understand why I'm seeing the columns defined as Strings in Python classified as VARCHAR in DB Browser for SQLite. If I attempt to modify the table, I see all of the expected datatypes for SQLite and also VARCHAR as an option. If I create a new database/table, then VARCHAR doesn't exist as an option for datatypes. Why wouldn't these columns be displayed as TEXT datatypes?
Strings in Python classified as VARCHAR in DB Browser for SQLite.
In Flask you are actually using SQLAlchemy ORM which will convert your class models directly into SQLite statements to create the relational database tables corresponding to it.
String data type in Python class model will be as VARCHAR data type in SQLite using Object Relational Mapper.
In SQLite, Text is the parent of VARCHAR and in the default installation is no different, so VARCHAR is actually same as TEXT.
Also If you check 3.1. Determination Of Column Affinity in the documentation you notice in the second point that:
If the declared type of the column contains any of the strings "CHAR",
"CLOB", or "TEXT" then that column has TEXT affinity. Notice that the
type VARCHAR contains the string "CHAR" and is thus assigned TEXT
affinity.
For more info check : http://www.sqlite.org/datatype3.html
I'm actually building a little utils which aims to take flat csv/excel file and populate a target database on MS Access - as I'm working on a Mac, i'm developping it using Postgres...
So I developped a part which deals with messy input (csv/excel) forms (several heading, etc) but that's not my issue at the moment.
On the other hand, I made my Database model using SQLAlchemy Declarative Base API.
I'm facing issue when importing data in some tables:
- Split flat record to several objects
- Check (SELECT) if the record doesn't exists yet based on uniqueness contraints
- If it doesn't exists I create object else I use the existing one
- Propagate keys information to related object
For some tables I'm using the auto_increment arguments but sometimes the record has its own ID (in input file) so I should it for insert/select in my tables and sometimes no ID so I have to create a new technical Id for my table.
Example: I have a record with for primary key -obsr25644- and sometimes nothing so I use a default value created with uuid.
So below the stacktrace when doing selectoperation on a my table. The same error occurs when working on existing data - obsr25644 - and generated uuid - 'a8098c1a-f86e-11da-bd1a-00112444be1e'
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidTextRepresentation) **invalid input syntax for integer**: "obsr25644"
LINE 3: WHERE "Location"."Id_observer" = 'obsr25644'
As you can see below, "Location"."Id_observer" is declared as String(255). I don't understand why the error is related to 'integer'.
[SQL: SELECT "Location"."Id_location" AS "Location_Id_location", [...], "Location"."Id_observer" AS "Location_Id_observer",
FROM "Location"
WHERE "Location"."Id_observer" = %(Id_observer_1)s
LIMIT %(param_1)s]
[parameters: {'Id_observer_1': 'obsr25644', 'param_1': 1}]
class LocationModel(UniqueMixin, Base):
__tablename__ = 'Location'
# Primary key
Id_location = Column(Integer, primary_key=True, autoincrement=True)
[...]
Id_observer = Column(String(255), ForeignKey('Observer.Id_observer'))
observer = relationship("ObserverModel", load_on_pending=True, back_populates="location")
class ObserverModel(UniqueMixin, Base):
__tablename__ = 'Observer'
# Primary key
Id_observer = Column(String(255), primary_key=True, default=UniqueMixin.unique_hash())
[...]
# Relationship
location = relationship("LocationModel", load_on_pending=True, back_populates="observer")
Note :UniqueMixin.unique_hash() returns uuid.uuid4().hex
If I have a SQLAlchemy declarative model like below:
class Test(Model):
__tablename__ = 'tests'
id = Column(Integer, Sequence('test_id_seq'), primary_key=True)
...
Atest_id = Column(Integer, ForeignKey('Atests.id'), nullable=True)
Btest_id = Column(Integer, ForeignKey('Btests.id'), nullable=True)
Ctest_id = Column(Integer, ForeignKey('Ctests.id'), nullable=True)
Dtest_id = Column(Integer, ForeignKey('Dtests.id'), nullable=True)
Etest_id = Column(Integer, ForeignKey('Etests.id'), nullable=True)
...
date = Column(DateTime)
status = Column(String(20)) # pass, fail, needs_review
And I would like to ensure that only one of the *test_id foreign keys is present in a given row, how might I accomplish that in SQLAlchemy?
I see that there is an SQLAlchemy CheckConstraint object (see docs), but MySQL does not support check constraints.
The data model has interaction outside of SQLAlchemy, so preferably it would be a database-level check (MySQL)
Well, considering your requisites "The data model has interaction outside of SQLAlchemy, so preferably it would be a database-level check (MySQL)" and 'ensure that only one [..] is not null'. I think the best approach is to write a trigger like this:
DELIMITER $$
CREATE TRIGGER check_null_insert BEFORE INSERT
ON my_table
FOR EACH ROW BEGIN
IF CHAR_LENGTH(CONCAT_WS('', NEW.a-NEW.a, NEW.b-NEW.b, NEW.c-NEW.c)) = 1 THEN
UPDATE `Error: Only one value of *test_id must be not null` SET z=0;
END IF;
END$$
DELIMITER ;
Some tricks and considerations:
IF STATEMENT: In order to avoid the tedious writing of check each column is not null while others are null, I did this trick: Reduce each column to one character and check how many characters exist. Note that NEW.a-NEW.a always returns 1 character if NEW.a is an Integer, NULL returns 0 characters and the operation NULL-NULL returns NULL on MySQL.
ERROR TRIGGERING: I suppose you want to raise an error, so how to do this on MySQL? You didn't mention the MySQL version. Only on MySQL 5.5 you can use the SIGNAL syntax to throw an exception. So the more portable way is issuing an invalid statement like: UPDATE xx SET z=0. If you are using MySQL 5.5 you could use: signal sqlstate '45000' set message_text = 'Error: Only one value of *test_id must be not null'; instead of UPDATE `Error: Only one value of *test_id must be not null` SET z=0;
Also, I think you want to check this on updates too, so use:
DELIMITER $$
CREATE TRIGGER check_null_update BEFORE UPDATE
ON my_table
FOR EACH ROW BEGIN
IF CHAR_LENGTH(CONCAT_WS('', NEW.a-NEW.a, NEW.b-NEW.b, NEW.c-NEW.c)) = 1 THEN
UPDATE `Error: Only one value of *test_id must be not null` SET z=0;
END IF;
END$$
DELIMITER ;
Or create a stored procedure and call it.
Update
For databases that supports check constraints, the code is more simple, see this example for SQL Server:
CREATE TABLE MyTable (col1 INT NULL, col2 INT NULL, col3 INT NULL);
GO
ALTER TABLE MyTable
ADD CONSTRAINT CheckOnlyOneColumnIsNull
CHECK (
LEN(CONCAT(col1-col1, col2-col2, col3-col3)) = 1
)
GO
I am accessing Postgre database using SQLAlchemy models. In one of models I have Column with UUID type.
id = Column(UUID(as_uuid=True), default=uuid.uuid4(), nullable=False, unique=True)
and it works when I try to insert new row (generates new id).
Problem is when I try to fetch Person by id I try like
person = session.query(Person).filter(Person.id.like(some_id)).first()
some_id is string received from client
but then I get error LIKE (Programming Error) operator does not exist: uuid ~~ unknown.
How to fetch/compare UUID column in database through SQLAlchemy ?
don't use like, use =, not == (in ISO-standard SQL, = means equality).
Keep in mind that UUID's are stored in PostgreSQL as binary types, not as text strings, so LIKE makes no sense. You could probably do uuid::text LIKE ? but it would perform very poorly over large sets because you are effectively ensuring that indexes can't be used.
But = works, and is far preferable:
mydb=>select 'd796d940-687f-11e3-bbb6-88ae1de492b9'::uuid = 'd796d940-687f-11e3-bbb6-88ae1de492b9';
?column?
----------
t
(1 row)