Develop Reference

(sqlite3.OperationalError) unknown database "seller"

I'm trying to use a testdb(sqlite) to make my tests, but when i use Base.metadata.createall() to create the same tables of the production database, i got this error: (sqlite3.OperationalError) unknown database "seller".
Conftest.py:
DATABASE_URL = 'sqlite:///testedb.sqlite'
#pytest.fixture(scope="function")
def client() -> Generator:
config_database(DATABASE_URL)
with TestClient(app) as c:
yield c
Database.py:
Base = declarative_base()
def config_database(database_url):
engine = create_engine(database_url)
Base.metadata.create_all(bind=engine)
Example of model there i'm using:
class Seller(Base):
__table__ = Table(
"seller",
Base.metadata,
Column(
"seller_id",
Integer,
primary_key=True,
index=True,
nullable=False),
Column("cnpj", String, nullable=True),
Column("nickname", String, nullable=False),
schema="seller")

Some database back-ends like PostgreSQL and MS SQL Server support the notion of a database containing multiple schemas, each of which can contain tables, views, stored procedures, etc.. If we are connected to a database named "my_db" then
SELECT * FROM seller.thing
means 'select rows from the table named "thing" in the schema named "seller" in the current database (my_db)'.
Other database back-ends like MySQL and SQLite do not support schemas within a database. Instead, they treat "schema" and "database" as synonyms, so
SELECT * FROM seller.thing
means 'select rows from the table named "thing" in the database named "seller", regardless of the current database'.
Therefore,
from sqlalchemy import create_engine, Column, Integer, Table, MetaData
engine = create_engine("sqlite:///data.db")
thing = Table(
"thing",
MetaData(),
Column("id", Integer, primary_key=True, autoincrement=False),
schema="seller",
)
engine.echo = True
thing.create(engine)
will fail with the error
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unknown database seller
[SQL:
CREATE TABLE seller.thing (
id INTEGER NOT NULL,
PRIMARY KEY (id)
)
]
if the current SQLite database does not have an attached database named "seller".
That might be a bit confusing because in the above example the database "data.db" will be created automatically if it does not exist, but that happens when the code tries to establish a (DBAPI) connection to the database. The same "auto-create" behaviour does not occur when an SQL statement tries to refer to another database.
So, if you want to use a "schema" named "seller" in SQLite then you need to ATTACH it to the current database like so:
from sqlalchemy import create_engine, Column, event, Integer, Table, MetaData
engine = create_engine("sqlite:///data.db")
#event.listens_for(engine, "first_connect")
def schema_attach(dbapi_connection, connection_record):
dbapi_connection.execute("ATTACH DATABASE 'seller.db' AS seller")
thing = Table(
"thing",
MetaData(),
Column("id", Integer, primary_key=True, autoincrement=False),
schema="seller",
)
engine.echo = True
thing.create(engine)
(Note that in this case "seller.db" will be automatically created if it does not exist.)

Get last inserted value from MySQL using SQLAlchemy

I've just run across a fairly vexing problem, and after testing I have found that NONE of the available answers are sufficient.
I have seen various suggestions but none seem to be able to return the last inserted value for an auto_increment field in MySQL.
I have seen examples that mention the use of session.flush() to add the record and then retrieve the id. However that always seems to return 0.
I have also seen examples that mention the use of session.refresh() but that raises the following error: InvalidRequestError: Could not refresh instance ''
What I'm trying to do seems insanely simple but I can't seem to figure out the secret.
I'm using the declarative approach.
So, my code looks something like this:
class Foo(Base):
__tablename__ = 'tblfoo'
__table_args__ = {'mysql_engine':'InnoDB'}
ModelID = Column(INTEGER(unsigned=True), default=0, primary_key=True, autoincrement=True)
ModelName = Column(Unicode(255), nullable=True, index=True)
ModelMemo = Column(Unicode(255), nullable=True)
f = Foo(ModelName='Bar', ModelMemo='Foo')
session.add(f)
session.flush()
At this point, the object f has been pushed to the DB, and has been automatically assigned a unique primary key id. However, I can't seem to find a way to obtain the value to use in some additional operations. I would like to do the following:
my_new_id = f.ModelID
I know I could simply execute another query to lookup the ModelID based on other parameters but I would prefer not to if at all possible.
I would much appreciate any insight into a solution to this problem.
Thanks for the help in advance.

The problem is you are setting defaul for the auto increment. So when it run the insert into query the log of server is
2011-12-21 13:44:26,561 INFO sqlalchemy.engine.base.Engine.0x...1150 INSERT INTO tblfoo (`ModelID`, `ModelName`, `ModelMemo`) VALUES (%s, %s, %s)
2011-12-21 13:44:26,561 INFO sqlalchemy.engine.base.Engine.0x...1150 (0, 'Bar', 'Foo')
ID : 0
So the output is 0 which is the default value and which is passed because you are setting default value for autoincrement column.
If I run same code without default then it give the correct output.
Please try this code
from sqlalchemy import create_engine
engine = create_engine('mysql://test:test#localhost/test1', echo=True)
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
from sqlalchemy.orm import sessionmaker
Session = sessionmaker(bind=engine)
session = Session()
from sqlalchemy import Column, Integer, Unicode
class Foo(Base):
__tablename__ = 'tblfoo'
__table_args__ = {'mysql_engine':'InnoDB'}
ModelID = Column(Integer, primary_key=True, autoincrement=True)
ModelName = Column(Unicode(255), nullable=True, index=True)
ModelMemo = Column(Unicode(255), nullable=True)
Base.metadata.create_all(engine)
f = Foo(ModelName='Bar', ModelMemo='Foo')
session.add(f)
session.flush()
print "ID :", f.ModelID

Try using session.commit() instead of session.flush(). You can then use f.ModelID.

Not sure why the flagged answer worked for you. But in my case, that does not actually insert the row into the table. I need to call commit() in the end.
So the last few lines of code are:
f = Foo(ModelName='Bar', ModelMemo='Foo')
session.add(f)
session.flush()
print "ID:", f.ModelID
session.commit()

How to write alter column name migrations with sqlalchemy-migrate?

I'm trying to alter a column name. First attempt was with this script:
meta = MetaData()
users = Table('users', meta,
Column('id', Integer, primary_key=True),
Column('name', String(50), unique=True),
Column('email', String(120), unique=True)
)
def upgrade(migrate_engine):
meta.bind = migrate_engine
users.c.id.alter(name='id')
def downgrade(migrate_engine):
meta.bind = migrate_engine
users.c.id.alter(name='user_id')
Running migrate.py test on my dev database (sqlite) works and so does upgrading and downgrading. But when deploying it to my test environment on Heroku (where PostgreSQL 8.3 is used) I get a trace when I try to upgrade. Gist is this message:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "id" does not exist
I then tried to use users.c.user_idin the upgrade method. That fails in both environments.:
AttributeError: user_id
The workaround I'm using now is this script:
meta_old = MetaData()
meta_new = MetaData()
users_old = Table('users', meta_old,
Column('user_id', Integer, primary_key=True),
Column('name', String(50), unique=True),
Column('email', String(120), unique=True)
)
users_new = Table('users', meta_new,
Column('id', Integer, primary_key=True),
Column('name', String(50), unique=True),
Column('email', String(120), unique=True)
)
def upgrade(migrate_engine):
meta_old.bind = migrate_engine
users_old.c.user_id.alter(name='id')
def downgrade(migrate_engine):
meta_new.bind = migrate_engine
users_new.c.id.alter(name='user_id')
It's already recommended practice to copy-paste the model to the sqlalchemy-migrate scripts. But this extra duplications gets a bit too much for me. Anyone knows how this should be done. Assuming it's a bug, I'd like suggestions on how to DRY up the workaround some.

This one also works:
from alembic import op
....
def upgrade(migrate_engine):
op.alter_column('users', 'user_id', new_column_name='id')
def downgrade(migrate_engine):
op.alter_column('users', 'id', new_column_name='user_id')

Turns out there's an even DRY:er solution to this than I had hoped for. Introspection! Like so:
def upgrade(migrate_engine):
meta = MetaData(bind=migrate_engine)
users = Table('users', meta, autoload=True)
users.c.user_id.alter(name='id')
def downgrade(migrate_engine):
meta = MetaData(bind=migrate_engine)
users = Table('users', meta, autoload=True)
users.c.id.alter(name='user_id')
Works like a charm!

I bet that it can't generate any SQL because your metadata references are getting mixed up. You seem to be using two different metadata objects in your Table classes, and that's really not good. You only need one. The metadata tracks stale-ness of objects, whether it needs to issue queries for object updates, foreign key constraints, etc. and it needs to know about all your tables and relationships.
Change to use a single MetaData object, and pass echo=True to your sqlalchemy.create_engine call and it will print the SQL query that it's using to standard output. Try executing that query yourself while logged in as the same role (user) to Postgres. You may find that it's a simple permissions issue.
Regarding copy-pasting: I think Django has a good convention of placing Table and declarative classes in their own module and importing them. However, because you have to pass a MetaData object to the Table factory, that complicates matters. You can use a singleton/global metadata object, or just convert to declarative.
For a while I chose to implement one-argument functions that returned Table objects given a metadata and cached the result--in effect implementing a singleton model class. Then I decided that was silly and switched to declarative.

sqlalchemy - how to manage this scenario - cyclic dependency

I have a database package which contains some modules namely student_table and a db module,
in student_table I have definition it's definition like this
from sqlalchemy import Table, MetaData, String, Column, Integer
metadata = MetaData()
class User(object):
def __init__(self, user_id, name):
self.user_id = user_id
self.name = name
user_table = Table('twitter_user', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(100))
)
and in db.py I have database create functions, which like this
def prepareDB():
"""
sets global variables based to access database
"""
read_settings()
engine = create_engine('mysql://'+setting_data["database_username"]+':'+setting_data["database_password"]+'#'+setting_data["database_host"]+'/'+setting_data["database_name"]+'?charset=utf8')
Session = sessionmaker(bind=engine)
global session
session = Session()
metadata.create_all(engine)
mapper(database.User, database.user_table)
my problem is metadata is required by db module to create engine and also by student module for definition however I don't see a way of doing so without creating cyclic dependency.
what can I do rectify this situation.

I have used something equivalent to this.
holder = [None]
def getmetadata():
if holder[0] is None:
holder[0] = sqlalchemy.MetaData()
return holder[0]
user_tablef = lambda metadata: Table('twitter_user', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(100))
)
# ...
user_table = user_tablef(getmetadata())
But this is not the cleanest thing in the world, and it's one reason a lot of people use the declarative style.

I just moved constant to separate file , now everything works fine.

I Keep receiving an "The MetaData is not bound to an Engine or Connection." when trying to create my SQL tables with pylons

Here is my current code:
def init_model(engine):
global t_user
t_user = sa.Table("User", meta.metadata,
sa.Column("id", sa.types.Integer, primary_key=True),
sa.Column("name", sa.types.String(100), nullable=False),
sa.Column("first_name", sa.types.String(100), nullable=False),
sa.Column("last_name", sa.types.String(100), nullable=False),
sa.Column("email", sa.types.String(100), nullable=False),
sa.Column("password", sa.types.String, nullable=False),
autoload=True,
autoload_with=engine
)
orm.mapper(User, t_user)
meta.Session.configure(bind=engine)
meta.Session = orm.scoped_session(sm)
meta.engine = engine
I then try to execute:
>>> meta.metadata.create_all(bind=meta.engine)
And receive the error:
raise exc.UnboundExecutionError(msg)
sqlalchemy.exc.UnboundExecutionError: The MetaData is not bound to an Engine or Connection. Execution can not proceed without a database to execute against. Either execute with an explicit connection or assign the MetaData's .bind to enable implicit execution.
In my development.ini I have:
# SQLAlchemy database URL
sqlalchemy.url = sqlite:///%(here)s/development.db
I'm new to Python's pylons and have no idea how to resolve this message. This is probably an easy fix to the trained eye. Thank you.

This issue was resolved. I didn't know that when using pylons from the CLI, I have to include the entire environment:
from paste.deploy import appconfig
from pylons import config
from project.config.environment import load_environment
conf = appconfig('config:development.ini', relative_to='.')
load_environment(conf.global_conf, conf.local_conf)
from project.model import *
After this the database queries executed without a problem.