Develop Reference

How to use the same code for both sqlite and postgres

My sqlalchemy code needs to support both sqlite and postgres, but right now it is not working for sqlite.
sqlalchemy.exc.StatementError: (builtins.TypeError) SQLite DateTime type only accepts Python datetime and date objects as input.
I checked Error - "SQLite DateTime type only accepts Python " "datetime and date objects as input." but making this change in my entire code base is not possible as it has more than one place where a date string is used instead of datetime
This is my code, it works for postgres engine, but it does not work for sqlite, can I modify anything other than what the above link suggests so my code runs on both sqlite and postgres
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import declarative_base, Session
from sqlalchemy.types import DateTime
Base = declarative_base()
class Foo(Base):
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
col = Column(DateTime)
engine = create_engine("postgresql://tony:tony#localhost:5432")
# engine = create_engine("sqlite:///db.db")
Base.metadata.create_all(engine)
with Session(engine) as session:
session.add(Foo(col='2023-01-07T11:08:31Z'))
session.commit()

The DB-API spec expects SQL DATETIME to be provided as python datetime.datetime (docs).
I believe psycopg2 offers an extenstion that will handle ISO 8601 formatted strings, but this is an extension.
If you want to be most compatible, use datetime.datetime object to pass and retrieve dates.
Also why import DateTime from sqlalchemy.types ? It's available under sqlalchemy directly.
from datetime import datetime
from sqlalchemy import Column, Integer, create_engine, DateTime
from sqlalchemy.orm import Session, declarative_base
Base = declarative_base()
class Foo(Base):
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
col = Column(DateTime)
engine = create_engine("postgresql+psycopg2://postgres:postgres#localhost:5432/postgres") # OK
engine = create_engine("sqlite:///db.db") # OK
Base.metadata.create_all(engine)
with Session(engine) as session:
session.add(Foo(col=datetime.fromisoformat("2023-01-07T11:08:31Z")))
session.commit()
If you need compatibility for runtime (not tests), you can use the following TypeDecorator.
Look further into the docs if you want more than coercing bind parameters, but this will allow your to input ISO 8601 str in SQLite.
from datetime import datetime
from sqlalchemy import Column, Integer, create_engine, DateTime
from sqlalchemy.orm import Session, declarative_base
from sqlalchemy.types import TypeDecorator
Base = declarative_base()
class ISO8601DateTime(TypeDecorator):
impl = DateTime
cache_ok = True
def process_bind_param(self, value, dialect):
if dialect.name == "sqlite" and isinstance(value, str):
value = datetime.fromisoformat(value)
return value
class Foo(Base):
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
col = Column(ISO8601DateTime) # NOTE: decorated type
engine = create_engine("postgresql+psycopg2://postgres:postgres#localhost:5432/postgres") # OK
engine = create_engine("sqlite:///db.db") # OK
Base.metadata.create_all(engine)
with Session(engine) as session:
session.add(Foo(col=datetime.fromisoformat("2023-01-07T11:08:31Z")))
session.add(Foo(col="2023-01-07T11:08:31Z"))
session.commit()
NB. for tests, if unit testing, mock the database adapter, if integration (and higher) testing, use the same DB as you'll use in production.

How to declare this column with SQLAlchemy

I have a MySQL table where one of the columns is a decimal. It is created with the following parameters:
`ratio` decimal(5,3) DEFAULT NULL,
So far the sample scripts I'm putting together is super dumbed down.
import sys
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()
class DailyProjections(Base):
__tablename__ = 'daily_projections'
id = Column(Integer, primary_key=True)
name = Column(String(100))
How do I declare the ratio column here?

From the docs:
The Numeric type is designed to receive data from a database type that
is explicitly known to be a decimal type (e.g. DECIMAL, NUMERIC,
others) and not a floating point type (e.g. FLOAT, REAL, others).
So you declare it like so:
ratio = Column(Numeric(5, 3), nullable=True)

Using Flask-SQLAlchemy without Flask

I had a small web service built using Flask and Flask-SQLAlchemy that only held one model. I now want to use the same database, but with a command line app, so I'd like to drop the Flask dependency.
My model looks like this:
class IPEntry(db.Model):
id = db.Column(db.Integer, primary_key=True)
ip_address = db.Column(db.String(16), unique=True)
first_seen = db.Column(db.DateTime(),
default = datetime.datetime.utcnow
)
last_seen = db.Column(db.DateTime(),
default = datetime.datetime.utcnow
)
#validates('ip')
def validate_ip(self, key, ip):
assert is_ip_addr(ip)
return ip
Since db will no longer be a reference to flask.ext.sqlalchemy.SQLAlchemy(app), how can I convert my model to use just SQLAlchemy. Is there a way for the two applications (one with Flask-SQLAlchemy the other with SQLAlchemy) to use the same database?

you can do this to replace db.Model:
from sqlalchemy import orm
from sqlalchemy.ext.declarative import declarative_base
import sqlalchemy as sa
base = declarative_base()
engine = sa.create_engine(YOUR_DB_URI)
base.metadata.bind = engine
session = orm.scoped_session(orm.sessionmaker())(bind=engine)
# after this:
# base == db.Model
# session == db.session
# other db.* values are in sa.*
# ie: old: db.Column(db.Integer,db.ForeignKey('s.id'))
# new: sa.Column(sa.Integer,sa.ForeignKey('s.id'))
# except relationship, and backref, those are in orm
# ie: orm.relationship, orm.backref
# so to define a simple model
class UserModel(base):
__tablename__ = 'users' #<- must declare name for db table
id = sa.Column(sa.Integer,primary_key=True)
name = sa.Column(sa.String(255),nullable=False)
then to create the tables:
base.metadata.create_all()

That is how to use SQLAlchemy without Flask (for example to write a bulk of objects to PostgreSQL database):
from sqlalchemy import Column, Integer, String
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
# Define variables DB_USERNAME, DB_PASSWORD, DB_HOST, DB_PORT, DB_NAME
SQLALCHEMY_DATABASE_URI = f'postgresql://{DB_USERNAME}:{DB_PASSWORD}#{DB_HOST}:
{DB_PORT}/{DB_NAME}'
# ----- This is related code -----
engine = create_engine(SQLALCHEMY_DATABASE_URI, echo=True)
Base = declarative_base()
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine)
session = Session()
# ----- This is related code -----
class MyModel(Base):
__tablename__ = 'my_table_name'
id = Column(Integer, primary_key=True)
value = Column(String)
objects = [MyModel(id=0, value='a'), MyModel(id=1, value='b')]
session.bulk_save_objects(objects)
session.commit()

Check this one github.com/mardix/active-alchemy
Active-Alchemy is a framework agnostic wrapper for SQLAlchemy that makes it really easy to use by implementing a simple active record like api, while it still uses the db.session underneath. Inspired by Flask-SQLAlchemy

There is a great article about Flask-SQLAlchemy: how it works, and how to modify models to use them outside of Flask:
http://derrickgilland.com/posts/demystifying-flask-sqlalchemy/

The sqlalchemy docs has a good tutorial with examples that sound like what you want to do.
Shows how to connect to a db, mapping, schema creation, and querying/saving to the db.

This does not completely answer your question, because it does not remove Flask dependency, but you can use SqlAlchemy in scripts and tests by just not running the Flask app.
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import MetaData
test_app = Flask('test_app')
test_app.config['SQLALCHEMY_DATABASE_URI'] = 'database_uri'
test_app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
metadata = MetaData(schema='myschema')
db = SQLAlchemy(test_app, metadata=metadata)
class IPEntry(db.Model):
pass
One difficulty you may encounter is the requirement of using db.Model as a base class for your models if you want to target the web app and independent scripts using same codebase. Possible way to tackle it is using dynamic polymorphism and wrap the class definition in a function.
def get_ipentry(db):
class IPEntry(db.Model):
pass
return IPEntry
As you construct the class run-time in the function, you can pass in different SqlAlchemy instances. Only downside is that you need to call the function to construct the class before using it.
db = SqlAlchemy(...)
IpEntry = get_ipentry(db)
IpEntry.query.filter_by(id=123).one()

Flask (> 1.0) attempt to provide helpers for sharing code between an web application and a command line interface; i personally think it might be cleaner, lighter and easier to build libraries unbound to flask, but you might want to check:
https://flask.palletsprojects.com/en/2.1.x/cli/
https://flask.palletsprojects.com/en/2.1.x/api/#flask.Flask.cli

Create database and table
import os
from sqlalchemy import create_engine
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
if os.path.exists('test.db'):
os.remove('test.db')
Base = declarative_base()
class Person(Base):
__tablename__ = 'person'
id = Column(Integer(), primary_key=True)
name = Column(String())
engine = create_engine('sqlite:///test.db')
Base.metadata.create_all(engine)
Using Flask_SQLAlchemy directly
from flask import Flask
from sqlalchemy import MetaData
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import Column, Integer, String
app = Flask(__name__)
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db = SQLAlchemy(app, metadata=MetaData())
class Person(db.Model):
__tablename__ = 'person'
id = Column(Integer(), primary_key=True)
name = Column(String())
person = Person(name='Bob')
db.session.add(person)
db.session.commit()
print(person.id)

SQLAchemy is never able to locate specified column in row

I'm new to SQLAlchemy (but not ORM's in general) and running into an issue where I literally cannot do anything useful. All I'm able to do is hard code my query, which is what I want to do.
Here is exactly what I'm attempting to do (see Python comments to see what does/doesn't work -
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String
from sqlalchemy import MetaData, Table
import pypyodbc
connection_string = 'mssql+pyodbc://U:P#SERVER/DB'
Base = declarative_base()
class Item(Base):
__tablename__ = 'Items'
id = Column('Id', Integer, primary_key=True)
engine = create_engine(connection_string, echo=True, module=pypyodbc)
sm = sessionmaker(engine)
session = sm()
# WORKS
for row in engine.execute('select top 1 Id from Items'):
print(row)
# DOES NOT WORK
print(session.query(Item).get(111))
The error I'm getting is sqlalchemy.exc.NoSuchColumnError: "Could not locate column in row for column 'Items.Id'".
The generated SQL is correct -
SELECT [Items].[Id] AS [Items_Id]
FROM [Items]
WHERE [Items].[Id] = 111
I know I'm pointing to the right database, because if I change __tablename to something non-existent I get an invalid object name error.
Why can't SQLAlchemy find my columns?

SQLAlchemy - create table template

I am just starting with SQLAlchemy and I have been wondering... I am going to have a lot of tables in my model. I would like to have own file for each table I will have in my model.
I am currently using following code:
from sqlalchemy import MetaData
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.dialects import postgresql
import sqlalchemy as sa
__all__ = ['Session', 'engine', 'metadata']
# SQLAlchemy database engine. Updated by model.init_model()
engine = None
# SQLAlchemy session manager. Updated by model.init_model()
Session = scoped_session(sessionmaker())
# Global metadata. If you have multiple databases with overlapping table
# names, you'll need a metadata for each database
metadata = MetaData()
# declarative table definitions
Base = declarative_base()
Base.metadata = metadata
schema = 'srbam_dev'
in meta.py
Following in _init_.py
"""The application's model objects"""
import sqlalchemy as sa
from sqlalchemy import orm
from models import meta
from models.filers import Filer
from models.vfilers import Vfiler
from models.filer_options import FilerOption
def init_models(engine):
"""Call me before using any of the tables or classes in the model"""
## Reflected tables must be defined and mapped here
#global reflected_table
#reflected_table = sa.Table("Reflected", meta.metadata, autoload=True,
# autoload_with=engine)
#orm.mapper(Reflected, reflected_table)
#
meta.engine = sa.create_engine(engine)
meta.Session.configure(bind=meta.engine)
class Basic_Table(object):
id = sa.Column(
postgresql.UUID(),
nullable=False,
primary_key=True
)
created = sa.Column(
sa.types.DateTime(True),
nullable=False
)
modified = sa.Column(
sa.types.DateTime(True),
nullable=False
)
And then following in all of my models
from models.meta import Base
from models.meta import Basic_Table
from models.meta import schema
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
class Filer(Base,Basic_Table):
This works just fine, as long as I do not start to use some Foreign Keys on tables, once I use Foreign Key says
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 't_vfilers.filer_id' could not find table 't_filers' with which to generate a foreign key to target column 'id'
I tried to define id key directly in Filer class (and remove Basic_Table from declaration), however this does not solve the issue.
My code for creating the database looks like this
#!/usr/bin/python
import ConfigParser
from sqlalchemy.engine.url import URL
from models import *
config = ConfigParser.RawConfigParser()
config.read('conf/core.conf')
db_url = URL(
'postgresql+psycopg2',
config.get('database','username'),
config.get('database','password'),
config.get('database','host'),
config.get('database','port'),
config.get('database','dbname')
)
init_models(db_url)
meta.Base.metadata.drop_all(bind=meta.engine)
meta.Base.metadata.create_all(bind=meta.engine)
Does anyone have an idea how to fix this issue?

Marek,do one thing try defining the foreign key along with the schema name i.e 'test.t_vfilers.filer_id'(here 'test' is the schema name),this will solve the problem.

Have you remembered to import the different modules containing the models.
In my init.py I have at the bottom a ton of:
from comparty3.model.users import User, UserGroup, Permission
from comparty3.model.pages import PageGroup, Page
etc...
If that's not the issue then I'm not sure; however have you tried to change:
metadata = MetaData()
# declarative table definitions
Base = declarative_base()
Base.metadata = metadata
to:
# declarative table definitions
Base = declarative_base()
metadata = Base.metadata
I'm guessing here, but it may be that declarative_base() creates a special metadata object.
This is how it is defined in my pylons projects (were I'm guessing your code is from too).