Django has the convenience manage.py command dumpdata which can be configured to dump an entire database as JSON.
At present I am confined to using sqlalchemy and I'd like to do the same thing:
Take as input a connection string like 'mysql+pymysql://user:pwd#localhost:3306/' and get the contents of the database as JSON (I don't require all the meta info that Django provides, but I won't mind).
I found this question elaborating how to dump SQLAlchemy objects to JSON and this from the sqlalchemy documentation outlining how to get all tablenames from a database:
meta = MetaData()
input_db = f'sqlite:///tmpsqlite'
engine = create_engine(input_db)
meta.reflect(bind=engine)
print(meta.tables)
How do I retrieve all of the content of these tables and then convert them to JSON? Is there a built-in command in sqlalchemy similar to django's dumpdata functionality?
Leaving my solution here for posterity:
import json
def dump_sqlalchemy(output_connection_string,output_schema):
""" Returns the entire content of a database as lists of dicts"""
engine = create_engine(f'{output_connection_string}{output_schema}')
meta = MetaData()
meta.reflect(bind=engine) # http://docs.sqlalchemy.org/en/rel_0_9/core/reflection.html
result = {}
for table in meta.sorted_tables:
result[table.name] = [dict(row) for row in engine.execute(table.select())]
return json.dumps(result)
Related
I'm trying to get familiar with SQLAlchemy, namely the ORM module. I've created a number of different table classes; TableClass1, TableClass2, TableClass3, etc. They all have a unique ID column. I'm using a MSSQL database.
I wanna to retrieve the max ID for each table, so I go ahead and do the following:
query1 = db.query(TableClass1.ID).order_by(TableClass1.ID.desc()).limit(1)
query2 = db.query(TableClass2.ID).order_by(TableClass2.ID.desc()).limit(1)
query3 = db.query(TableClass3.ID).order_by(TableClass3.ID.desc()).limit(1)
Connection settings are
engine = create_engine(
f'mssql+pyodbc://{SERVER}/{DATABASE}?{DRIVER}')
SessionLocal = sessionmaker(bind=engine)
db = SessionLocal()
QUESTION
However, this is sort of trivial. Is it possible to create some sort of callable method, where I can pass the names of the tables dynamically? I was thinking of something like:
def query_func(tablename):
query = db.query(tablename.ID).order_by(tablename.ID.desc()).limit(1)
return query
I am new to flask and python in general. I just need help accessing an already existing snowflake database with flask. I just want to query the data. This is my code thus far and its not working:
from flask import Flask, render_template, request, redirect, url_for
from model import InputForm
from compute import preprocess
from sqlalchemy import create_engine, MetaData, Table
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABSE_URI'] = 'snowflake://<user_login_name>:<password>#<account_name>'
db = SQLAlchemy(app)
engine = create_engine()
metadata = MetaData(bind=engine)
engagements = db.Table('ENGAGEMENTS', db.metadata, autoload=True, autoload_with=db.engine)
companies = db.Table('COMPANIES', db.metadata, autoload=True, autoload_with=db.engine)
#app.route('/')
def index():
results = db.session.query(engagements).all()
for r in results:
print(engagements)
return ''
if __name__ == '__main__':
app.run(debug=True)
There may be multiple things going on here.
First, if the error message you get is due to non-existent engagements table in the Snowflake db, it may be exactly that. Before you can execute queries, you need to bring your database at par with the models in your code. https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-iv-database is a great source on how to set up your db. It teaches how to set up a normal sqlite database for migrations using Alembic. Luckily snowflake-sqlalchemy has alembic support: https://github.com/snowflakedb/snowflake-sqlalchemy#alembic-support . You can use them to build and export migrations, and your db should be ready.
Second, there's two typos: one in your code, one in the documentation. Your code sets app.config['SQLALCHEMY_DATABSE_URI'] instead of app.config['SQLALCHEMY_DATABASE_URI']. And you also definitely need to mention your db and schema for snowflake as pointed out in https://stackoverflow.com/a/59204661/2928486. But the official documentation has incorrect uri.
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>?role=<role_name>'
Instead of the above, it should be the following:
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>&role=<role_name>'
Basically your connection params should be joined using & instead of ?. I was able to set up my flask application using your code, and it is migrating and updating the snowflake db fine, so hopefully it works for other ddl/dql scenarios as well!
I'm not a flask user, but it is possible you need to include the database and schema. I found a URL example below that includes snowflake account, database, schema, warehouse and role.
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>?role=<role_name>'
I'm trying to get the number of rows from a SQL Server which consists of many tables by looping through it to see how much data they contain. However, I'm not sure what will go into select_from() function. As I currently supply Unicode for table names and it raised
NoInspectionAvailable: No inspection system is available for object of type <type 'unicode'>
The code that I used was
from sqlalchemy import create_engine
import urllib
from sqlalchemy import inspect
import sqlalchemy
from sqlalchemy import select, func, Integer, Table, Column, MetaData
from sqlalchemy.orm import sessionmaker
connection_string = "DRIVER={SQL Server}; *hidden*"
connection_string = urllib.quote_plus(connection_string)
connection_string = "mssql+pyodbc:///?odbc_connect=%s" % connection_string
engine = sqlalchemy.create_engine(connection_string)
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
connection = engine.connect()
inspector = inspect(engine)
for table_name in inspector.get_table_names():
print session.query(func.count('*')).select_from(table_name).scalar()
Typically, it's a class name that refers to a class that describes the database table.
In the sqlalchemy docs, http://docs.sqlalchemy.org/en/latest/orm/tutorial.html, they have you create a base class using declarative base and then create child classes for each table you want to query. You would then pass that class name into the select_from function unquoted.
The Flask framework provides a built-in base class that is ready for use called db.Model and Django has one called models.Model.
Alternatively, you can also pass queries. I use the Flask framework typically for python so I usually initiate queries like this:
my_qry = db.session.query(Cust).filter(Cust, Cust.cust == 'lolz')
results = my_qry.all()
On a side note, if you decide to look at .NET they also have nice ORMs. Personally, I favor Entity Framework, but Linq to SQL is out there, too.
I'm using a flask setup from a while and now trying to install Flask-Blogging module on it. Current modules:
- Flask-sqlalchemy with postgres
- Flask-login
- Flask-Blogging (new)
My application.py looks like this:
from flask import Flask
from flask import session
from flask.ext.blogging import SQLAStorage, BloggingEngine
from flask.ext.login import LoginManager
from flask.ext.sqlalchemy import SQLAlchemy
'''
The main application setup. The order of things is important
in this file.
'''
app = Flask(__name__)
app.config.from_object('config.base')
app.config.from_envvar('APP_CONFIG_FILE')
'''
Initialize database
'''
db = SQLAlchemy(app)
'''
Initialize blogger
'''
storage = SQLAStorage(db=db)
blog_engine = BloggingEngine(app, storage)
the last two lines are the only new things I added (other than the imports). Suddenly now I'm getting error about duplicate table names:
sqlalchemy.exc.InvalidRequestError: Table 'customer' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Any ideas what am I doing wrong? I couldn't find much documentation about Flask-Blogging other than:
http://flask-blogging.readthedocs.org/en/latest/
You get this error because in SQLAStorage.__init__ there is this line:
self._metadata.reflect(bind=self._engine)
This will look at your database and create sqlalchemy table things for all the tables currently in your database.
Thus if your database contains a table called 'customer' the line in your code:
storage = SQLAStorage(db=db)
will automatically model an sqlalchemy table called 'customer' for you.
Now... no doubt you have your own database model definitions somewhere, probably in another python module, something like:
class Customer(db.Model):
id = db.Column(db.Integer, primary_key=True)
...
Since this class definition defines a table called 'customer' and since SQLAStorage has already defined a table called 'customer' you get the exception as soon as your class Customer is imported.
Some ways to work around this problem are:
Import your database definition modules before instantiating SQLAStorage
'''
Initialize database
'''
db = SQLAlchemy(app)
import ankit.db_models # import my db models here so SQLAStorage doesn't do it first
'''
Initialize blogger
'''
storage = SQLAStorage(db=db)
blog_engine = BloggingEngine(app, storage)
or
Tell SQLAStorage to use its own metadata
By passing the db param to SQLAStorage.__init__ you are telling it to use your metadata. You can instead just pass the engine parameter and it will create its own metadata.
storage = SQLAStorage(engine=db.engine)
I have a custom management command to import data from one db and create model instances from it. Basically it is:
class Command(BaseCommand):
def handle(self, **kwargs):
cursor = connections['db2'].cursor()
sql = 'SELECT * FROM table'
cursor.execute(sql)
for row in cursor.fetchall():
my_model = MyModel(*row)
my_model.save()
First I was importing from sqlite to sqlite and all went well. But when I switched to MySQL as my main db, I started getting UnicodeEncodeError, when calling my_model.save(). The caveat is that I have non-ascii symbols in db (namely Russian), but as I get, Django converts all strings to unicode. And yes, both dbs use utf-8.
It seems that bug is due to mezzanine app, I'm using
https://github.com/stephenmcd/mezzanine/issues/1132