I've been using sqlalchemy to do my rest implementation and now I want to get a list of tables from the database. I have tried these:
# first attempt
engine = create_engine(datasource.connection_string)
insp = reflection.Inspector.from_engine(engine)
tables = insp.get_table_names()
view = insp.get_view_names()
# second attempt
meta = MetaData()
meta.reflect(bind=engine, views=True)
While they both work perfectly fine they both have their downsides.
The first does not give me the schema but only the object name
The second gives me the world but runs dog slow...
For both there is no filtering
Using a sql-statement is not an option since I need it to be somewhat cross-db.
Is there a way to quickly load the objects including schema? Filtering is of less importance since I can do that on the list of objects fast enough.
get_table_names() and meta.reflect() only load tables from one schema at a time. If an explicit schema name is not given, the "default" schema is assumed, which is the schema that your database connection automatically selects when you connect; this is configured on the database side in conjunction with the user account used to connect.
if you want to see the name of this schema, use the "default_schema_name" property:
>>> from sqlalchemy import create_engine, inspect
>>> e = create_engine("postgresql://scott:tiger#localhost/test")
>>> insp = inspect(e)
>>> insp.default_schema_name
u'public'
Related
I have two tables X and Y.
X has two columns id and name,
Y has three columns id, city, country
I need to make a join query and take city and country values corresponding to its name using id. id is a foreign key.
I tried directly in PostgreSQL with this query,
select * from x inner join y on x.name = 'xyz'
It's giving exact result. But while trying the same query using sqlalchemy orm in Python flask app it's not. I'm new to Python and Flask framework. I don't know how to achieve it.
I declared table definition and serialize function in a separate python file inside my models directory.
In my main.py file, I'm importing that model and tried this kind of querying, But it's not working.
from models import x, y
response = x.query.join(y).filter_by(name=name).all()
I'm getting error message like this,
sqlalchemy.exc.InvalidRequestError
InvalidRequestError: Could not find a FROM clause to join from. Tried joining to <class 'models.y'>, but got: Can't find any foreign key relationships between 'x' and 'y'.
If you prefer to write out regular SQL statements, then why not use SQLAlchemy to do just that. It's not a requirement that you use chain methods like join, filter_by, etc to query your database.
In order to answer this question, I need to make some assumptions about what's in your models file. I'll assume it's something like this:
from flask_sqlalchemy import SQLAlchemy
import datetime
db = SQLAlchemy()
class BaseModel(db.Model):
"""Base data model for all objects"""
# more code here
class x(BaseModel, db.Model):
# more table setup code here
class y(BaseModel, db.Model):
# more table setup code here
If that's the case, then here's what you can do to execute plain old parameterized SQL statements::
from flask import Flask
from models import db
import json
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'your_database_connection_string'
db.init_app(app)
result = db.session.execute("select * from x inner join y on x.name = :name", {"name":"xyz"})
# If no rows were returned in the result, return an empty list
if result.returns_rows == False:
response = []
# Convert the response to a plain list of dicts
else:
response = [dict(row.items()) for row in result]
# Output the query result as JSON
print(json.dumps(response))
I find this method of running SQL queries in Flash with SQLAlchemy far easier to follow and understand than trying to use all of the different method chaining that you were attempting in your original post.
Although it is possible to avoid the ORM alltogether like in Elliot's example, the ORM does have its advantages if the database is small. The most convenient way to solve your problem is by adding a relationship. You can then query the results by, for example:
tree = db.session.query(Tree).first()
for leaf in tree.leaves:
print(leaf.id)
If you cannot create the relationship, the proper syntax for a join is as follows:
db.session.query(Leaf).join(Tree,Leaf.tree_id==Tree.id)
I'm wondering if there's a way to merge new mappings with database data, such as with session.merge, but without updating the database? Like when I do a pull with git, to get a local state which is a merge of the remote and previous local state(which might contain unpushed commits), but without updating the remote state. Here, I want to get a local "view" of the state that would result from doing a session.merge.
Maybe making savepoint(with session.begin_nested), then doing a session.merge and later on a session.rollback would accomplish this, but is there a way that doesn't require this kind of transaction management(which can imply actual undo operations on the db, so not terribly efficient for my purposes)?
Would using session.no_autoflush do the trick?
Example code for what I want to do:
local_merge = session.merge_local(Model(...))
# do stuff with local_merge...
remotes = session.query(Model).all()
# remotes should remain "old" db versions, as no data was pushed
return something
Edit: So I think I may be wrong on the rollback method being inefficient. At least, as long as no commit are emitted, there shouldn't be expensive undo operations, only chucking out the transaction.
Merge will only update the database because of the auto-flush. You can turn that off temporarily using the session.no_autoflush context manager, or just setting autoflush=False on your session. You can also pass autoflush=False to your sessionmaker.
One thing to watch out for is how the results of the session.query(Model).all() will interact with the unflushed, changed, local objects.
Because the session maintains a record of unique objects (against primary keys) in an Identity Map, you will not be able to have two versions of the same object in the same session.
Here's an example which shows how local changes interact with autoflush=False:
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
def __repr__(self):
return "<User(name='%s')>" % self.name
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
ed_user = User(name='ed')
session.add(ed_user)
session.commit()
ed_again = session.query(User).get(1)
ed_again.name = "not ed"
with session.no_autoflush:
ed_three = session.query(User).get(1)
all_eds = session.query(User).all()
print(ed_again, id(ed_again))
print(ed_three, id(ed_three))
print(all_eds, id(all_eds[0]))
<User(name='not ed')> 139664151068624
<User(name='not ed')> 139664151068624
[<User(name='not ed')>] 139664151068624
Yep, it's not able to get the original Ed from the database, even with no_autoflush - this is to be expected for get(), since it will check the identity map first before the database, and won't bother querying the DB if it finds it in the identity map. But with query.all(), it queries the database, finds that one of the objects it gets back was already in the identity map, and returns that reference instead so as to maintain uniqueness of objects in the session (which was also my hunch, but I couldn't find this explicitly spelled out in the docs).
You could do something like expunge objects from a session, but I think the easiest way to have an old and new copy of the merged objects is to use two separate sessions, one where the changes will be merged and possibly committed and one which you can use to check the existing state of objects in the database.
I am using the core of Sqlalchemy so I am not using a declarative base class like in other similar questions.
How to get the primary key of a table using the engine?
So, I just ran into the same problem. You need to create a Table object that reflects the table for which you are looking to find the primary key.
from sqlalchemy import create_engine, Table, MetaData
dbUrl = 'postgres://localhost:5432/databaseName' #will be different for different db's
engine = create_engine(dbUrl)
meta = MetaData()
table = Table('tableName', meta, autoload=True, autoload_with=engine)
primaryKeyColName = table.primary_key.columns.values()[0].name
The Table construct above is useful for a number of different functions. I use it quite a bit for managing geospatial tables since I do not really like any of the current packages out there.
In your comment you mention that you are not defining a table...I think that means that you aren't creating a sqlalchemy model of the the table. With the approach above, you don't need to do that and can gather all sorts of information from a table in a more dynamic fashion. Especially useful if you are be handed someone else's messy database!
Hope that helps.
I'd like to comment, but I do not have enough reputation for that.
Building on greenbergé answer:
from sqlalchemy import create_engine, Table, MetaData
dbUrl = 'postgres://localhost:5432/databaseName' #will be different for different db's
engine = create_engine(dbUrl)
meta = MetaData()
table = Table('tableName', meta, autoload=True, autoload_with=engine)
[0] is not always the PK, only if the PK has only one column.
table.primary_key.columns.values() is a list.
In order to get all columns of a multi-column pk you could use:
primaryKeyColNames = [pk_column.name for pk_column in table.primary_key.columns.values()]
The two answers were given for retrieving the primary key from a metadata object.
Even if it works well, sometimes one can look for retrieving the primary key from an instance of a SQL Alchemy model, without even knowing what actually the model class is (for example if you search for having a helper function called, let's say, get_primary_key, that would accept an instance of a DB Model and output the primary keys).
For this we can use the inspect function from the inspection module :
from sqlalchemy import inspect
def get_primary_key(model_instance):
inspector = inspect(model_instance)
model_columns = inspector.mapper.columns
return [c.description for c in model_columns if c.primary_key]
You could also directly use the __mapper__ object
def get_primary_key(model_instance):
model_columns = model_instance.__mapper__.columns
return [c.description for c in model_columns if c.primary_key]
for a reflected table this works:
insp=inspect(self.db.engine)
pk_temp=insp.get_pk_constraint(self.__tablename__)['constrained_columns']
I'm using python sqlite3 api to create a database.
In all examples I saw on the documentation table names and colum names are hardcoded inside queries..but this could be a potential problem if I re-use the same table multiple times (ie, creating table, inserting records into table, reading data from table, alter table and so on...) because In case of table modification I need to change the hardcoded names in multiple places and this is not a good programming practice..
How can I solve this problem?
I thought creating a class with just constructor method in order to store all this string names..and use it inside the class that will operation on database..but as I'm not an expert python programmer I would like to share my thoughts...
class TableA(object):
def __init__(self):
self.table_name = 'tableA'
self.name_col1 = 'first_column'
self.type_col1='INTEGER'
self.name_col2 = 'second_column'
self.type.col2 = 'TEXT'
self.name_col3 = 'third_column'
self.type_col3 = 'BLOB'
and then inside the DB classe
table_A = TableA()
def insert_table(self):
conn = sqlite3.connect(self._db_name)
query = 'INSERT INTO ' + table_A.table_name + ..... <SNIP>
conn.execute(query)
Is this a proper way to proceed?
I don't know what's proper but I can tell you that it's not conventional.
If you really want to structure tables as classes, you could consider an object relational mapper like SQLAlchemy. Otherwise, the way you're going about it, how do you know how many column variables you have? What about storing a list of 2-item lists? Or a list of dictionaries?
self.column_list = []
self.column_list.append({'name':'first','type':'integer'})
The way you're doing it sounds pretty novel. Check out their code and see how they do it.
If you are going to start using classes to provide an abstraction layer for your database tables, you might as well start using an ORM. Some examples are SQLAlchemy and SQLObject, both of which are extremely popular.
Here's a taste of SQLAlchemy:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
Base = declarative_base()
class TableA(Base):
__tablename__ = 'tableA'
id = Column(Integer, primary_key=True)
first_column = Column(Integer)
second_column = Column(String)
# etc...
engine = create_engine('sqlite:///test.db')
Base.metadata.bind = engine
session = sessionmaker(bind=engine)()
ta = TableA(first_column=123, second_column='Hi there')
session.add(ta)
session.commit()
Of course you would choose semantic names for the table and columns, but you can see that declaring a table is something along the lines of what you were proposing in your question, i.e. using a class. Inserting records is simplified by creating instances of that class.
I personally don't like to use libraries and frameworks without proper reason. So, if I'd such reason, so will write a thinking wrapper around sqlite.
class Column(object):
def __init__(self, col_name="FOO", col_type="INTEGER"):
# standard initialization
And then table class that encapsulates operations with database
class Table(object):
def __init__(self, list_of_columns, cursor):
#initialization
#create-update-delete commands
In table class you can encapsulate all operations with the database you want.
I'm using SA 0.6.6, Declarative style, against Postgres 8.3, to map Python objects to a database. I have a table that is self referencing and I'm trying to make a relationship property for it's children. No matter what I try, I end up with a NoReferencedTableError.
My code looks exactly like the sample code from the SA website for how to do this very thing.
Here's the class.
class FilterFolder(Base):
__tablename__ = 'FilterFolder'
id = Column(Integer,primary_key=True)
name = Column(String)
isShared = Column(Boolean,default=False)
isGlobal = Column(Boolean,default=False)
parentFolderId = Column(Integer,ForeignKey('FilterFolder.id'))
childFolders = relationship("FilterFolder",
backref=backref('parentFolder', remote_side=id)
)
Here's the error I get:
NoReferencedTableError: Foreign key assocated with column 'FilterFolder.parentFolderId' could not find table 'FilterFolder' with which to generate a foreign key to target column 'id'
Any ideas what I'm doing wrong here?
This was a foolish mistake on my part. I typically specify my FK's by specifying the Entity type, not the string. I am using different schemas, so when defining the FK entity as a string I also need the schema.
Broken:
parentFolderId = Column(Integer,ForeignKey('FilterFolder.id'))
Fixed:
parentFolderId = Column(Integer,ForeignKey('SchemaName.FilterFolder.id'))
I checked your code with SQLAlchemy 0.6.6 and sqlite. I was able to create the tables, add a parent and child combination, and retrieve them again using a session.query.
As far as I can tell, the exception you mentioned (NoReferencedTableError) is thrown in schema.py (in the SQLAlchemy source) exclusively, and is not database specific.
Some questions: Do you see the same bug if you use an sqlite URL instead of the Postgres one? How are you creating your schema? Mine looks something like this:
engine = create_engine(db_url)
FilterFolder.metadata.create_all(self.dbengine)