TL;DR: How do I use a metadata object from a Blueprint to create the Flask-SQLAlchemy instance? The only place I can see to provide the declarative base metadata object is in the initial SQLAlchemy() call. But when I import it from the Blueprint in my extensions.py file, the Blueprint's code needs the db object and the loading fails due to a circular import.
I have several model classes which I would like to use both in and outside of Flask. I am using the declarative method to do this, and my app is set up to use the App Factory model and Blueprints. The way the models get registered with SQLAlchemy is through the use of the metadata argument when the db object is created. In the context of my application, it makes sense to declare the metadata object in a Blueprint, instead of in the main app Blueprint. (This is where most of the code which references it will be, including the non-Flask utility script used for initially populating the database.) However, importing the model class from the second Blueprint ends up in a circular import.
$ flask db migrate
Error: While importing "my_app", an ImportError was raised:
Traceback (most recent call last):
File "my_app/venv/lib/python3.7/site-packages/flask/cli.py", line 235, in locate_app
__import__(module_name)
File "my_app/my_app.py", line 1, in <module>
from app import create_app
File "my_app/app/__init__.py", line 7, in <module>
from app.extensions import *
File "my_app/app/extensions.py", line 10, in <module>
from turf.models import metadata
File "my_app/turf/__init__.py", line 1, in <module>
from .routes import bp
File "my_app/turf/routes.py", line 14, in <module>
from app.extensions import db
ImportError: cannot import name 'db' from 'app.extensions' (my_app/app/extensions.py)
As mentioned in this general question on circular imports for Blueprints, a solution which works is to import the db object from inside each function in the second Blueprint, thus side-stepping the import during initialization of the extensions.py file. But in addition to being annoying, this just feels extremely hacky.
Ideally, I would be able to pass the metadata object I've created to SQLAlchemy's init_app() method. That would resolve this issue at a stroke. Unfortunately, init_app() doesn't take a metadata argument. Is there some other way to register metadata with an SQLAlchemy instance after initialization? Or have I missed some other key element of the declarative model approach?
I should say that the non-Flask portion of this is working just fine. My utility script is able to import the models and use them to add objects to the database. It's only the Flask imports which are giving me trouble.
Here is the hierarchy:
.
├── app
│ ├── __init__.py
│ └── extensions.py
└── turf
├── __init__.py
├── models.py
└── routes.py
And the relevant code which fails due to circular import:
app/__init__.py:
from app.extensions import *
def create_app():
app = Flask(__name__)
with app.app_context():
import turf
app.register_blueprint(turf.bp)
db.init_app(app)
app/extensions.py:
from turf.models import metadata
db = SQLAlchemy(metadata=metadata)
turf/__init__.py:
from .routes import bp
turf/models.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import MetaData
metadata = MetaData()
Base = declarative_base(metadata=metadata)
# All the turf models are declared in this file
class Boundary(Base):
# ...etc...
turf/routes.py:
from .models import *
from app.extensions import db
bp = Blueprint('Turf', __name__, url_prefix='/turf')
#bp.route('/')
def index():
return render_template('turf/index.html')
It turns out you can declare the MetaData object in the extensions.py file and then import that in the Blueprint. I was sure this would fail, because the metadata object is now being populated after the db object is created, but I've verified that the models are indeed available and work as expected. And no more circular dependency. I've actually broken this part out into its own file to allow Blueprint code to import as little as possible.
app/base.py:
from sqlalchemy import MetaData
from sqlalchemy.ext.declarative import declarative_base
metadata = MetaData()
Base = declarative_base(metadata=metadata)
app/extensions.py:
from flask_sqlalchemy import SQLAlchemy
from .base import metadata
db = SQLAlchemy(metadata=metadata)
turf/models.py:
from app.base import Base
# All the turf models are declared in this file
class Boundary(Base):
# ...etc...
This also answers another question I had with my original approach: how would it work if I had a second Blueprint which also had model objects which needed to be available from non-Flask code? Now, I have created a single Base object and can use that to implement new classes in different Blueprints as needed.
One minor annoyance with this approach, though. In the non-Flask DB population script, I was originally able to use from models import * to refer to the sibling module (file) containing the models. This let me call the script directly, à la cd turf; python populate_db.py --arg. That no longer works because the models.py file now references a different package, app.extensions. So instead I have to use this workaround:
turf/populate_db.py:
try:
from .models import *
except:
print("You have to run this from outside the 'turf' directory like so: $ python -m turf.populate_db [...]")
sys.exit(1)
Related
I've had a look at a very similar issue, but the solution isn't working for me, and I was wondering if I could get some guidance.
This is my application structure:
mosiman/
__init__.py
blog/
__init__.py
blogconfig.py
blog.py
static/
...
templates/
...
I really wanted a way to build separate apps but manage them from one application instance so I wouldn't have to touch the WSGI layer, and blueprints seemed like the best way.
Here's what I think the relevant files are:
# mosiman/__init__.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
import blog.blog as blog
...
...
db = SQLAlchemy()
...
...
def create_app():
app = Flask(__name__)
db_path = os.path.join(os.path.dirname(__file__), 'mosiman.db')
app.config["SQLALCHEMY_DATABASE_URI"] = 'sqlite:///{}'.format(db_path)
app.config["SQLALCHEMY_BINDS"] = {'blog': os.path.join(os.path.dirname(__file__), 'blog/blog.db')
db.init_app(app)
app.register_blueprint(blog.blog, url_prefix='/blog')
return app
if __name__ == '__main__':
app = create_app()
app.run(debug=True)
I also have in my mosiman/blog/blog.py file:
#mosiman/blog/blog.py
<<< all of the usual flask imports and SQLAlchemy, etc >>>
from .__init__ import db
blog = Blueprint('blog', __name__)
...
...
class Entry(db.Model):
__bind_key__ = "blog"
id = db.Column(db.Integer, primary_key = True)
title = db.Column(db.String(150))
...
...
Here is my problem: I actually don't have a "default" database, so I have no classes for it. But I do want to initialize my blog database with the class Entry.
The default procedure for initializing the database seems to be to pop into the python REPL, import the database, and run db.create_all()
Well, I can't just import db and run db.create_all() because db hasn't been attached to an app yet (done in create_app()) This is how I proceed:
# pop open a REPL at mosiman/
>>> from __init__ import db
>>> from __init__ import create_app
>>> app = create_app()
>>> db.init_app(app)
>>> from blog.blog import Entry
>>> db.init_app(app)
# At this point, db still isn't attached to an engine?
>>> db
<SQLAlchemy engine=None>
>>> with app.app_context():
... db.create_all()
...
>>>
Now, when I go to mosiman/blog/ I find that blog.db has been created, but when I check the schema and tables in SQLite3 nothing turns up.
1 solved their problem by changing from __init__ import db to from mosiman import db
If I try to do that inside mosiman/:
>>> from mosiman import db
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mosiman'
even though I've got an __init__.py in there.
If I run it from ../ (the directory above mosiman):
>>> from mosiman import db
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/users/d49chan/mosiman/__init__.py", line 3, in <module>
import blog.blog as blog
ModuleNotFoundError: No module named 'blog'
I'm not sure how to proceed. Maybe it's because I need sleep, but I've been trying to understand why nothing works for the past many hours. If you could point me in the right direction or possibly clear up some of my misunderstandings I'd be very grateful.
1) You don't namely import __init__.py. You should import it with the name of the directory (package) it's in.
2) You shouldn't import like this: import blog.blog. Instead you should: from blog import blog.
3) Be sure you have an __init__.py file in every directory so that the python importer can treat it as a package (directory) it can import from.
4) There's a difference between relative importing and absolute importing.
Relative importing:
# l3_module.py in l3_package
from ..l1_package import l1_module
This imports a module located two packages (directories) up in relation to the module importing it.
Now assuming l3_package is in an l2_package which is in an l1_module, You can also import it as such
Absolute importing:
# l3_moudle.py in l3_package
from l1_package import l1_module
This will also work, but then it sometimes messes stuff up, depending on your env.
Here's an excellent answer that can clear some stuff up for you: Relative imports for the billionth time.
Hope that helped :)
I want to separate my model classes into separate files in a models directory. I would like to have a separate file for:
general (authentication and global classes/tables)
requisitions (tables used for requisitions)
workorders (tables used for workorders)
sales_orders (tables used for sales orders)
...etc
I'm not sure how to structure my project to make that happen.
I've tried putting my main imports into init.py in the directory and them importing those into the individual model files, but I don't know where to put my db.generate_mapping() so that all classes are available. I'm guessing this is a best practice for a large application. I've got about 150 tables in my app at this point.
Any help/pointers would be appreciated.
You can use the following project structure:
# /myproject
# settings.py
# main.py
# /models
# __init__.py
# base.py
# requisitions.py
# workorders.py
# sales_orders.py
settings.py is a file with database settings:
# settings.py
db_params = {'provider': 'sqlite', 'filename': ':memory:'}
main.py is a file when you start application. You put db.generate_mapping here:
# main.py
from pony import orm
import settings
from models import db
from your_favorite_web_framework import App
# orm.set_sql_degug(True)
db.bind(**settings.db_params)
db.generate_mapping(create_tables=True)
with orm.db_session:
orm.select(u for u in db.User).show()
if __name__ == '__main__':
app = App()
app.run()
Note that it is not necessary to implicitly import all models, as they are accessible as attributes of db object (like db.User)
You can put db object in base.py (or general.py), where you define your core models:
# base.py
from pony import orm
db = orm.Database()
class User(db.Entity):
name = Required(str)
orders = Set('Order')
Note that in User model I can refer to Order model defined in another module. You can also write it as
orders = Set(lambda: db.Order)
Unfortunately, IDEs like PyCharm at this moment cannot recognize that db.Order refers to specific Order class. You can import and use Order class directly, but in some cases it will lead to problem with cyclic imports.
In other model files you import db from .base:
# workorders.py
from pony import orm
from .base import db
class Order(db.Entity):
description = Optional(str)
user = Required('User')
In /models/__init__.py you import all modules to ensure that all models classes are defined before generate_mapping is called:
# /models/__init__.py
from .base import db
from . import workorders
...
And then you can write
from models import db
And access models as db attributes like db.User, db.Order, etc.
If you want to refer models directly in your code instead of accessing them as db attributes, you can import all models in __init__.py:
# /models/__init__.py
from .base import db, User
from .workorders import Order
...
And then you can import model classes as:
from models import db, User, Order, ...
My project:
project_name
|- my_app
|- __init__.py
|- run.py
|- models.py
First example
run.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
from models import User
db.create_all()
if __name__ == '__main__':
app.run()
models.py
from run import db
class User(db.Model):
#...
__ init.py __ is empty
After running this example I get this error:
ImportError: cannot import name User
This error describe Circular import of app variable in models.py (as I understand).
Second example
run.py
from my_app import app
if __name__ == '__main__':
app.run()
models.py
from my_app import db
class User(db.Model):
#...
__ init.p __
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
from models import User
db.create_all()
And now all works correct.
And, at this point, I don't understand, why the same code give me different logic?
Where is the magic? Why Circular import in __ init.py __ does't throw an error?
Thank You!
I'm just going to talk about your first example, because you can actually reproduce both scenarios without changing the code:
Example 1: Run run.py directly to see the Import error
Example 2: Open a python repl, and run:
from run import app
app.run()
And your first example now works, for the same reason your second code example did: the imports were moved out of __main__
In example 1, run.py is the top level execution environment, and the code here runs in __main__. The relative import to models.py requires that all references in models.py be resolved (since the class you're importing from models.py may be dependent on other parts of models.py outside of the class itself). The first thing models.py does is go back to __main__ to perform a relative import of db, and since it needs to resolve all references, it does - and one of the references it tries to resolve is the original from models import user statement. Voila, ImportError.
The thing is, at this point run.py hasn't finished executing, and so far it's still trying to import the definition of User. But now models.py is trying to call into it as though it had finished loading, and it expects run.py to already know the definition of User, even though it's right in the middle of trying to figure that exact thing out!
This isn't a circular import - not really. models.py is just trying to import from a file that hasn't finished executing yet, and whose contents aren't available for use.
By moving __main__ elsewhere - into the repl, or by moving your code around so that run.py doesn't define anything - you're giving the code a chance to actually resolve all the dependencies before any code tries to use them. By using from run import app; app.run(), this is the only code that is not done executing. The imports themselves all get to complete their work, resolve their dependencies and make them available to __main__
tl;dr don't accidentally write code that imports from __main__ when __main__ doesn't yet know the things it's being asked for
I know this question has been asked many times, but even after I tried many SO suggestions, I still cannot get it right. So, posting here for your help.
I'm building a simple web server with Flask-Restful and Flask-SqlAlchemy. Basically, I think I'm getting an error because of a circular import. My Widget class is dependent on Visualization class and vice versa...
The error message:
Traceback (most recent call last):
File "server.py", line 1, in <module>
from app.resources.dashboard import Dashboards, Dashboard
File "app/__init__.py", line 14, in <module>
from models import *
File "app/models/__init__.py", line 1, in <module>
import visualization.Visualization as VisualizationModel
File "app/models/visualization.py", line 3, in <module>
from app.models import WidgetModel
ImportError: cannot import name WidgetModel
Directory structure:
├── app
│ ├── app/__init__.py
│ ├── app/models
│ │ ├── app/models/__init__.py
│ │ ├── app/models/dashboard.py
│ │ ├── app/models/visualization.py
│ │ ├── app/models/widget.py
│ └── app/resources
│ ├── app/resources/__init__.py
│ ├── app/resources/dashboard.py
│ ├── app/resources/visualization.py
│ ├── app/resources/widget.py
├── server.py
app/models/__init__.py:
from visualization import Visualization as VisualizationModel
from widget import Widget as WidgetModel
from dashboard import Dashboard as DashboardModel
app/models/visualization.py
from sqlalchemy import types
from app import db
from app.models import WidgetModel
class Visualization(db.Model):
__tablename__ = 'visualizations'
...
widget = db.relationship(WidgetModel, cascade="all, delete-orphan", backref="visualizations")
app/models/widget.py
from app import db
from app.models import VisualizationModel
class Widget(db.Model):
__tablename__ = 'widgets'
...
visualization = db.relationship(VisualizationModel, backref="widgets")
I tried changing my import to from app import models, and then use models.WidgetModel / models.VisualizationModel. However, still getting an ImportError.
Error message:
Traceback (most recent call last):
File "server.py", line 1, in <module>
from app.resources.dashboard import Dashboards, Dashboard
File "app/__init__.py", line 14, in <module>
from models import *
File "app/models/__init__.py", line 1, in <module>
from visualization import Visualization as VisualizationModel
File "app/models/visualization.py", line 3, in <module>
from app import models
ImportError: cannot import name models
I'm very new to Python. I would be grateful if you can help me out. Thanks for you help in advance!
Update
The intention of defining bi-directional relationship is that I want to attach the Visualization object in the fields of Widget object upon a return of GET request on a widget record.
In the app/resources/widget.py I have:
...
from flask_restful import fields, Resource, marshal_with, abort
from app.models import WidgetModel
import visualization as visualizationResource
widget_fields = {
'id': fields.String,
...
'dashboard_id': fields.String,
'visualization_id': fields.String,
'visualization': fields.Nested(visualizationResource.visualization_fields)
}
class Widgets(Resource):
#marshal_with(widget_fields)
def get(self):
return WidgetModel.query.all()
I also want to have the cascade delete feature because if a widget cannot exist without a visualization.
Change your import to look like this:
from app import models
And then use models.WidgetModel / models.VisualizationModel instead.
The problem is that you're creating a circular import dependency where both files require the other file to already have been processed before they can be processed. By moving to importing the entire models namespace rather than trying to import a specific model name at import time, you avoid making the import dependent on a fully processed file. That way, both files can be fully processed before either tries to invoke an object created by the other.
It may still not work in this case however because you're trying to immediately use the import in part of the class definition, which is evaluated at processing time.
It looks like you're trying to define bi-directional relationships - the backref parameter is intended to do this automatically without you having to specify the relationship on both models. (backref tells sqlalchemy what field to add to the other model to point back to the original models that link to it). So you may not need to be doing both of these imports in the first place.
For instance, the fact that Visualization.widget is defined with backref="visualizations" means that Widget.visualizations will already exist - you don't need to explicitly create it.
If what you specifically need is many:many relationships, then chances are what you actually want to do is define an association table for the many-to-many relationship.
I'm trying to import an object named db (SQLAlchemy object) in in a module of my Flask Project in order to use the SQLAlchemy in my models (models.py). Assuming my package is named Foo and contains the db object in his __init__.py file , when i try to do a from Foo import db, i get the following error:
ImportError: cannot import name db
I'm using Flask Blueprint to dispatch my project into two applications (dashboard and frontend) and each of them contains an __init__.pyfile. Only the models.py is throwing this error, i got some imports in my views file (as importing the login manager) and everything goes well.
Any idea of what it could be ?
UPDATE : The traceback
Traceback (most recent call last):
File "run.py", line 4, in <module>
from kuchiyose import app
File "/path_to_project/kuchiyose/kuchiyose/__init__.py", line 60, in <module>
from kuchiyose import dashboard, frontend
File "/path_to_project/kuchiyose/kuchiyose/dashboard/__init__.py", line 10, in <module>
from dashboard import views
File "/path_to_project/kuchiyose/kuchiyose/dashboard/__init__.py", line 10, in <module>
from dashboard import views
File "/path_to_project/kuchiyose/kuchiyose/dashboard/views.py", line 8, in <module>
from kuchiyose.dashboard.models import User
File "/path_to_project/kuchiyose/kuchiyose/dashboard/models.py", line 3, in <module>
from kuchiyose import db
ImportError: cannot import name db
First, thanks for the help. Concerning my second question : "How to import Flask models into views without having a circular import issue (when using SQLAlchemy)", i found a solution. It consists to setup the SQLAlchemy object not into the application __init__.pyfile but into the models.pyfile itself. With that, you can import it in your views.pyfile without problem.
My models.py file, now:
from kuchiyose import app
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
And my views.py
from kuchiyose.dashboard import models
This is an old one but showed up as a top result in my search.
I have a project that has model declarations in a few different places rather than a consolidated models.py file so this solution wouldn't be ideal. I found importing your views AFTER establishing the db object works as well.
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config.from_object('config')
db = SQLAlchemy(app)
# Now Register routes
# db is now defined so app import db will now work)
from routes.user import user as user_routes
app.register_blueprint(user_routes)
there appears to be circular imports:
kuchiyose/__init__.py imports from kuchiyose.dashboard.views
kuchiyose/dashboard/views.py imports from kuchiyose.dashboard.models
kuchiyose/dashboard/models.py imports from kuchiyose
see e.g. Circular (or cyclic) imports in Python for more info...