Develop Reference

Creating Flask-SQLAlchemy instance with metadata from Blueprint

TL;DR: How do I use a metadata object from a Blueprint to create the Flask-SQLAlchemy instance? The only place I can see to provide the declarative base metadata object is in the initial SQLAlchemy() call. But when I import it from the Blueprint in my extensions.py file, the Blueprint's code needs the db object and the loading fails due to a circular import.
I have several model classes which I would like to use both in and outside of Flask. I am using the declarative method to do this, and my app is set up to use the App Factory model and Blueprints. The way the models get registered with SQLAlchemy is through the use of the metadata argument when the db object is created. In the context of my application, it makes sense to declare the metadata object in a Blueprint, instead of in the main app Blueprint. (This is where most of the code which references it will be, including the non-Flask utility script used for initially populating the database.) However, importing the model class from the second Blueprint ends up in a circular import.
$ flask db migrate
Error: While importing "my_app", an ImportError was raised:
Traceback (most recent call last):
File "my_app/venv/lib/python3.7/site-packages/flask/cli.py", line 235, in locate_app
__import__(module_name)
File "my_app/my_app.py", line 1, in <module>
from app import create_app
File "my_app/app/__init__.py", line 7, in <module>
from app.extensions import *
File "my_app/app/extensions.py", line 10, in <module>
from turf.models import metadata
File "my_app/turf/__init__.py", line 1, in <module>
from .routes import bp
File "my_app/turf/routes.py", line 14, in <module>
from app.extensions import db
ImportError: cannot import name 'db' from 'app.extensions' (my_app/app/extensions.py)
As mentioned in this general question on circular imports for Blueprints, a solution which works is to import the db object from inside each function in the second Blueprint, thus side-stepping the import during initialization of the extensions.py file. But in addition to being annoying, this just feels extremely hacky.
Ideally, I would be able to pass the metadata object I've created to SQLAlchemy's init_app() method. That would resolve this issue at a stroke. Unfortunately, init_app() doesn't take a metadata argument. Is there some other way to register metadata with an SQLAlchemy instance after initialization? Or have I missed some other key element of the declarative model approach?
I should say that the non-Flask portion of this is working just fine. My utility script is able to import the models and use them to add objects to the database. It's only the Flask imports which are giving me trouble.
Here is the hierarchy:
.
├── app
│   ├── __init__.py
│   └── extensions.py
└── turf
   ├── __init__.py
   ├── models.py
   └── routes.py
And the relevant code which fails due to circular import:
app/__init__.py:
from app.extensions import *
def create_app():
app = Flask(__name__)
with app.app_context():
import turf
app.register_blueprint(turf.bp)
db.init_app(app)
app/extensions.py:
from turf.models import metadata
db = SQLAlchemy(metadata=metadata)
turf/__init__.py:
from .routes import bp
turf/models.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import MetaData
metadata = MetaData()
Base = declarative_base(metadata=metadata)
# All the turf models are declared in this file
class Boundary(Base):
# ...etc...
turf/routes.py:
from .models import *
from app.extensions import db
bp = Blueprint('Turf', __name__, url_prefix='/turf')
#bp.route('/')
def index():
return render_template('turf/index.html')

It turns out you can declare the MetaData object in the extensions.py file and then import that in the Blueprint. I was sure this would fail, because the metadata object is now being populated after the db object is created, but I've verified that the models are indeed available and work as expected. And no more circular dependency. I've actually broken this part out into its own file to allow Blueprint code to import as little as possible.
app/base.py:
from sqlalchemy import MetaData
from sqlalchemy.ext.declarative import declarative_base
metadata = MetaData()
Base = declarative_base(metadata=metadata)
app/extensions.py:
from flask_sqlalchemy import SQLAlchemy
from .base import metadata
db = SQLAlchemy(metadata=metadata)
turf/models.py:
from app.base import Base
# All the turf models are declared in this file
class Boundary(Base):
# ...etc...
This also answers another question I had with my original approach: how would it work if I had a second Blueprint which also had model objects which needed to be available from non-Flask code? Now, I have created a single Base object and can use that to implement new classes in different Blueprints as needed.
One minor annoyance with this approach, though. In the non-Flask DB population script, I was originally able to use from models import * to refer to the sibling module (file) containing the models. This let me call the script directly, à la cd turf; python populate_db.py --arg. That no longer works because the models.py file now references a different package, app.extensions. So instead I have to use this workaround:
turf/populate_db.py:
try:
from .models import *
except:
print("You have to run this from outside the 'turf' directory like so: $ python -m turf.populate_db [...]")
sys.exit(1)

Python - Error importing class from another file

Can anyone guide on resolving the error ? The manager class method will need capability to create instance of User Class but it is not finding the module
Below is the folder structure and code and other details. init.py file is empty
Internal folder
->main.py
->__init__.py
->User Folder
-> user.py
-> models.py
-> usermanager.py
-> __init__.py
ModuleNotFoundError: No module named 'models'
at <module> (/home/vmagent/app/internal/user/usermanager.py:4)
at <module> (/home/vmagent/app/main.py:2)
at import_app (/env/lib/python3.6/site-packages/gunicorn/util.py:352)
at load_wsgiapp (/env/lib/python3.6/site-
packages/gunicorn/app/wsgiapp.py:52)
at load (/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py:65)
at wsgi (/env/lib/python3.6/site-packages/gunicorn/app/base.py:67)
at load_wsgi (/env/lib/python3.6/site-
packages/gunicorn/workers/base.py:135)
at init_process (/env/lib/python3.6/site-
packages/gunicorn/workers/base.py:126)
at spawn_worker (/env/lib/python3.6/site-
packages/gunicorn/arbiter.py:578)
main.py code
from internal.account import user,usermanager
def registerBlueprint(app, blueprint, manager):
blueprint.manager = manager
app.register_blueprint(blueprint)
registerBlueprint(app, user.request, usermanager)
user.py code
request = Blueprint('flex.user', __name__)
#request.route('/v1/user', methods=['POST'])
def CreateAccount():
test = request.manager.UserManager()
return test.CreateUser()
usermanager.py
from models import User
class UserManager:
def __init__(self):
pass
def CreateUser(self):
logger.log_text("AccountManger")
data = json.dumps(dict(
email="Hello#123.com",
password="Test124"
))
user = User(data)
user.create()
responseObject = dict(status='success', message='Successfully
registered.')
return make_response(jsonify(responseObject)), 201

Python paths can get kind of tricky. I don't claim that this is the best approach, but it's what I hacked together on my lunch. Perhaps it can get you where you need to go.
When you run main.py, you're in the internal/ directory. You need to go up a level and then access the user/ directory. That's what happens with sys.path.append('../'). The module usermanager.py is in the user/ directory, so to import from user.py, we want to point to the current directory. Hence the period in from .user import User.
Here's the directory structure I used. I believe it is the same as what you have.
C:.
├───internal
│ main.py
│ __init__.py
│
└───user
│ models.py
│ user.py
│ usermanager.py
└───__init__.py
__init__.py
The __init__.py files are empty. Their presence indicates to Python that the directories are modules1.
main.py
import sys
sys.path.append('../')
from user import user
from user import usermanager
from user import models
my_user = user.User()
my_user.message()
my_user_manager = usermanager.UserManager()
my_user_manager.message()
my_user_manager.CreateUser()
my_model = models.Model()
my_model.message()
models.py
class Model():
def message(self):
print("This is a Model!")
user.py
class User():
def message(self):
print("This is a User!")
usermanager.py
from .user import User
class UserManager():
def message(self):
print("This is a UserManager!")
def CreateUser(self):
new_user = User()
print("I created the user: ", new_user)
Now, when I call main.py I get:
c:\project\internal>python main.py
This is a User!
This is a UserManager!
I created the user: <user.user.User object at 0x0000000002A61EB8>
This is a Model!
Everyone is talking to everyone else. Hopefully you can extrapolate that to your use case! If not, Chris Yeh has an article "The Definitive Guide to Python import Statements" you may find helpful.
1 See What is __init__.py for? to learn more about how that works.

There is no models module in the standard library, and you do not have a file named models.py listed here, so python cannot find it. If models.py is in some subdirectory, that subdirectory needs to be in your python path, and you need to be sure that they contain an __init__.py file. That is the meaning of the error you're encountering here.

Flask-SQLAlchemy creating SQLite database but not tables

I've had a look at a very similar issue, but the solution isn't working for me, and I was wondering if I could get some guidance.
This is my application structure:
mosiman/
__init__.py
blog/
__init__.py
blogconfig.py
blog.py
static/
...
templates/
...
I really wanted a way to build separate apps but manage them from one application instance so I wouldn't have to touch the WSGI layer, and blueprints seemed like the best way.
Here's what I think the relevant files are:
# mosiman/__init__.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
import blog.blog as blog
...
...
db = SQLAlchemy()
...
...
def create_app():
app = Flask(__name__)
db_path = os.path.join(os.path.dirname(__file__), 'mosiman.db')
app.config["SQLALCHEMY_DATABASE_URI"] = 'sqlite:///{}'.format(db_path)
app.config["SQLALCHEMY_BINDS"] = {'blog': os.path.join(os.path.dirname(__file__), 'blog/blog.db')
db.init_app(app)
app.register_blueprint(blog.blog, url_prefix='/blog')
return app
if __name__ == '__main__':
app = create_app()
app.run(debug=True)
I also have in my mosiman/blog/blog.py file:
#mosiman/blog/blog.py
<<< all of the usual flask imports and SQLAlchemy, etc >>>
from .__init__ import db
blog = Blueprint('blog', __name__)
...
...
class Entry(db.Model):
__bind_key__ = "blog"
id = db.Column(db.Integer, primary_key = True)
title = db.Column(db.String(150))
...
...
Here is my problem: I actually don't have a "default" database, so I have no classes for it. But I do want to initialize my blog database with the class Entry.
The default procedure for initializing the database seems to be to pop into the python REPL, import the database, and run db.create_all()
Well, I can't just import db and run db.create_all() because db hasn't been attached to an app yet (done in create_app()) This is how I proceed:
# pop open a REPL at mosiman/
>>> from __init__ import db
>>> from __init__ import create_app
>>> app = create_app()
>>> db.init_app(app)
>>> from blog.blog import Entry
>>> db.init_app(app)
# At this point, db still isn't attached to an engine?
>>> db
<SQLAlchemy engine=None>
>>> with app.app_context():
... db.create_all()
...
>>>
Now, when I go to mosiman/blog/ I find that blog.db has been created, but when I check the schema and tables in SQLite3 nothing turns up.
1 solved their problem by changing from __init__ import db to from mosiman import db
If I try to do that inside mosiman/:
>>> from mosiman import db
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mosiman'
even though I've got an __init__.py in there.
If I run it from ../ (the directory above mosiman):
>>> from mosiman import db
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/users/d49chan/mosiman/__init__.py", line 3, in <module>
import blog.blog as blog
ModuleNotFoundError: No module named 'blog'
I'm not sure how to proceed. Maybe it's because I need sleep, but I've been trying to understand why nothing works for the past many hours. If you could point me in the right direction or possibly clear up some of my misunderstandings I'd be very grateful.

1) You don't namely import __init__.py. You should import it with the name of the directory (package) it's in.
2) You shouldn't import like this: import blog.blog. Instead you should: from blog import blog.
3) Be sure you have an __init__.py file in every directory so that the python importer can treat it as a package (directory) it can import from.
4) There's a difference between relative importing and absolute importing.
Relative importing:
# l3_module.py in l3_package
from ..l1_package import l1_module
This imports a module located two packages (directories) up in relation to the module importing it.
Now assuming l3_package is in an l2_package which is in an l1_module, You can also import it as such
Absolute importing:
# l3_moudle.py in l3_package
from l1_package import l1_module
This will also work, but then it sometimes messes stuff up, depending on your env.
Here's an excellent answer that can clear some stuff up for you: Relative imports for the billionth time.
Hope that helped :)

PonyORM - multiple model files

I want to separate my model classes into separate files in a models directory. I would like to have a separate file for:
general (authentication and global classes/tables)
requisitions (tables used for requisitions)
workorders (tables used for workorders)
sales_orders (tables used for sales orders)
...etc
I'm not sure how to structure my project to make that happen.
I've tried putting my main imports into init.py in the directory and them importing those into the individual model files, but I don't know where to put my db.generate_mapping() so that all classes are available. I'm guessing this is a best practice for a large application. I've got about 150 tables in my app at this point.
Any help/pointers would be appreciated.

You can use the following project structure:
# /myproject
# settings.py
# main.py
# /models
# __init__.py
# base.py
# requisitions.py
# workorders.py
# sales_orders.py
settings.py is a file with database settings:
# settings.py
db_params = {'provider': 'sqlite', 'filename': ':memory:'}
main.py is a file when you start application. You put db.generate_mapping here:
# main.py
from pony import orm
import settings
from models import db
from your_favorite_web_framework import App
# orm.set_sql_degug(True)
db.bind(**settings.db_params)
db.generate_mapping(create_tables=True)
with orm.db_session:
orm.select(u for u in db.User).show()
if __name__ == '__main__':
app = App()
app.run()
Note that it is not necessary to implicitly import all models, as they are accessible as attributes of db object (like db.User)
You can put db object in base.py (or general.py), where you define your core models:
# base.py
from pony import orm
db = orm.Database()
class User(db.Entity):
name = Required(str)
orders = Set('Order')
Note that in User model I can refer to Order model defined in another module. You can also write it as
orders = Set(lambda: db.Order)
Unfortunately, IDEs like PyCharm at this moment cannot recognize that db.Order refers to specific Order class. You can import and use Order class directly, but in some cases it will lead to problem with cyclic imports.
In other model files you import db from .base:
# workorders.py
from pony import orm
from .base import db
class Order(db.Entity):
description = Optional(str)
user = Required('User')
In /models/__init__.py you import all modules to ensure that all models classes are defined before generate_mapping is called:
# /models/__init__.py
from .base import db
from . import workorders
...
And then you can write
from models import db
And access models as db attributes like db.User, db.Order, etc.
If you want to refer models directly in your code instead of accessing them as db attributes, you can import all models in __init__.py:
# /models/__init__.py
from .base import db, User
from .workorders import Order
...
And then you can import model classes as:
from models import db, User, Order, ...

Unable to run an individual test with Django 1.8

Having followed the advice in these two answers, I'm still unable to run only a single django test in the way it describes. I can run them all, but that takes a long time and makes debugging harder.
My project structure looks like this:
mainapp/mainapp/users/
├── __init__.py
├── tests
│ ├── __init__.py
│ ├── test_views.py
└── views.py
For example, when the whole test suite is run, one of the failing tests gives this output:
FAIL: test_changepassword (mainapp.users.tests.test_views.ChangePasswordTest)
But when I try to run that Django TestCase with either the Django...
~/mainapp$ python manage.py test mainapp.users.tests.test_views.ChangePasswordTest
File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_views'
or django-nose syntax...
~/mainapp$ python manage.py test mainapp.users.tests.test_views:ChangePasswordTest
File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_views:ChangePasswordTest'
When I run this in PyCharmCE debug mode it confirms that the module object has no attributes, but I don't know why.
I have django-nose (1.4.5) installed and it's in my INSTALLED_APPS, but I don't know if anything else is required to use it.
This works fine down to a certain level. For example this is OK:
~/mainapp$ python manage.py test mainapp.users.tests
What's going on?