Develop Reference

How do I add an endpoint to my Django backend to run a python script?

I currently have a Django backend deployed in Heroku, and I also have a Python script that contains code to return the items with the highest number of mentions in chosen subreddits. It is noted the items are returned in tuple form (item, number of mentions) in a list. I would like to integrate the Python script into the backend as an API endpoint, such that the frontend can call the Python script on demand and in particular, supply user-defined input into the Python script (eg: which subreddit to scrape). Hence, for this reason, runscript in Django-extensions will not be suitable. How should I go about achieving my aim? Any help will be greatly appreciated!
The main function in my Python script looks like this:
#reddit is a Reddit instance in praw
#subname is name of subreddit to scrape from
#active_stocks is a dictionary of currently active stocks
#set default time limit as "day" to return all posts within the day itself
#set default limit as None so that all posts are returned
def track_mentions_in_past_24_hours(reddit,subname,active_stocks,time_filter="day",limit=None):
#main body of code is omitted
#returns dictionary of stocks with key value pair as (stock,number of mentions)
return active_stocks

This can be handled using #api_view(['GET']).
You could wrap the function with a decorator and integrate it with URLs.
# Imports
from rest_framework.decorators import api_view
from rest_framework.response import Response
def track_mentions_in_past_24_hours(reddit,subname,active_stocks,time_filter="day",limit=None):
# ...
return active_stocks
#api_view(['GET'])
def script_runner(request):
data = track_mentions_in_past_24_hours(reddit,subname,active_stocks,time_filter="day",limit=None)
return Response(data)
And that's how you can set up the endpoint that will be callable from the front-end app.
urlpatterns = [
path('script/', views.script_runner),
]
For more details: Docs

url_for in celery doesn't work

We are building a notofication mailing system as part of a python/flask system, as for now it has been run using cronjobs but we're moving everything to celery to make it more performant and easier to maintain.
However the templates which have been working so far suddendly start throwing the following error:
[2017-05-29 20:30:30,411: WARNING/PoolWorker-7] [2017-05-29 20:30:30,411]
ERROR in mails: ERROR errorString => Could not build url for endpoint
'page.start' with values ['from_email']. Did you mean 'static' instead?
The url_for is called in an external template as follows:
{{ url_for('page.start', _external=True) }}
and rendered as follows:
message = render_template('notifs/user_notif.html',
subject=subject,
entries = grouped,
user=u,
unsubscribe_hash=pw_hash,
list_id = str(notif_list.id),
timestamp = today)
Now if we rip out all the url_for in the template it works. But why?
My hypothesis, which i can't test or proof: Somehow celery does not have access to the Blueprints (even though it is running in the application context, as the tasks actually accesses all kinds of models and the db etc.). How do I make celery understand url_for?

Just ran into the same issue:
The database and models are fronted by by your ORM (Flask-sqlalchemy?), not the Flask application itself. Flask the application provides the context for things like url_for, current_user, etc. Your orm just provides the database abstraction and is not contingent on the actual application context.

How to structure a CherryPy app into a MVC architecture?

I was planning for my structure to be something similar to this:
appname/libs/user
/football
/tax
Where the library user, will have the models for the User, the controller which would show up the REST JSON API and the views.
The problems that I'm facing currently can be divided into two major questions, and which mainly stem from using Django for a while. I'm fairly new to CherryPy and SqlAlchemy.
How to define modles in each of this library? The problem I face is I've to inherit the Base Declarative and Engine in every model and run it as a standalone app for its models to be generated. Is there a mechanism where I can plug in the libraries and the database should pull all the models and create it? (Something that Django does.)
How to define routes/apis? (a urls.py)

How about defining the declarative base (sqlalchemy) in appname/db/__init__.py and for each of the libs import the base from appname in appname/libs/NAME/models.py:
import appname.db
Base = appname.db.Base
class MyUser(Base):
...
To get a database session just use a scoped session for example, this could be the basic content for appname/db/__init__.py (or just db.py if you don't want to define additional base models in appname/db/models.py)
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy import engine_from_config
__all__ = ['session', 'ses']
ses = session = scoped_session(sessionmaker())
def load_engine(config):
engine = engine_from_config(config, prefix='')
session.configure(bind=engine)
Then set a tool to remove the session from the thread locals when the request has ended:
import cherrypy
from appname import db
def remove_db_session():
db.session.remove()
cherrypy.tools.removedbs = cherrypy.Tool('on_end_request', remove_db_session)
from that point forward just use the session as normal from any part of your application, for example:
from appname import db
from appname.libs.user import models
class User:
exposed = True
def GET(self, id):
db.ses.query(models.User).filter_by(id=id)
# and let the tool to remove the session
# when the request finish
By the way to enable that removedbs tool, just make sure that you execute somewhere that cherrypy.tools.removedbs = .... I usually put that in: appname/cptools/tool and then in the config dictionary or file set tools.removedbs.on to True
Working with cherrypy means that you will build the application tree, there is no extra magic you need to have a central place to build the full tree, if you want to use the MethodDispatcher or the DefaultDispatcher.
In this case I recommend you the MethodDispatcher, and probably this post can give you a little more perspective and this is from my blog emulating the github api without any base handler.
There is an alternative to use more django like routes with the RoutesDispatcher, but you will lose a lot of functionality from the tools.
To show you an example with the MethodDispatcher and building your own object tree, from the current structure you can have a build function on the appname/builder.py and make something like this:
from appname.views import Main
from appname.libs import user, football
appmap = {'user': user,
'footbal': football}
def build_app(*apps):
root = Main()
for app in apps:
appmodule = appmap[app]
appmodule.attach(root)
return root
And inside the appname/libs/user/__init__.py
from appname.libs.user import views
def build_tree():
root = views.Main()
root.management = views.Management()
return root
def attach(root):
root.user = build_tree()
That's just a way to structure the application, there is also a repository with cherrypy recipes which are pretty helpful.

How should an index.html file be served from Python Pyramid for an AngularJS app?

I am getting into single-page apps with AngularJS but rather than using Node or similar, I am most comfortable with Python on the server. So, given I am somewhat familiar with Pyramid, I plan to use the pyramid_rpc module to return JSON objects to the client app. That's all straight forward enough, however, what then is the best way to serve the starting index file which contains the AngularJS initial AngularJS app? Usually, static files are served from a static directory, but is there any problem serving a index.html file from the root? Or should I use a view-callable and '/' route with a renderer to a html template? All that being said, is Pyramid overkill for this kind of application? Any advice would be great.

If you're planning to return some JSON responses then Pyramid is a great option. But I wouldn't recommend using pyramid_rpc. JSON-RPC is a protocol that is intended for RPC communication between servers. Straight json responses fit most clients (like browsers) better such as just a bunch of routes that return JSON responses in response to GET/POST requests. This is also a good place to serve up index.html, probably with a good http_cache parameter to prevent clients from requesting that page too often (of course you can go further with optimizing this route, but you should probably save that for later).
config.add_route('index', '/')
config.add_route('api.users', '/api/users')
config.add_route('api.user_by_id', '/api/users/{userid}')
#view_config(route_name='index', renderer='myapp:templates/index.html', http_cache=3600*24*365)
def index_view(request):
return {}
#view_config(route_name='api.users', request_method='POST', renderer='json')
def create_user_view(request):
# create a user via the request.POST parameters
return {
'userid': user.id,
}
#view_config(route_name='api', request_method='GET', renderer='json')
def user_info_view(request):
userid = request.matchdict['userid']
# lookup user
return {
'name': user.name,
}

Using Flask-SQLAlchemy in Blueprint models without reference to the app [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I'm trying to create a "modular application" in Flask using Blueprints.
When creating models, however, I'm running into the problem of having to reference the app in order to get the db-object provided by Flask-SQLAlchemy. I'd like to be able to use some blueprints with more than one app (similar to how Django apps can be used), so this is not a good solution.*
It's possible to do a switcharoo, and have the Blueprint create the db instance, which the app then imports together with the rest of the blueprint. But then, any other blueprint wishing to create models need to import from that blueprint instead of the app.
My questions are thus:
Is there a way to let Blueprints define models without any awareness of the app they're being used in later -- and have several Blueprints come together? By this, I mean having to import the app module/package from your Blueprint.
Am I wrong from the outset? Are Blueprints not meant to be independent of the app and be redistributable (à la Django apps)?
If not, then what pattern should you use to create something like that? Flask extensions? Should you simply not do it -- and maybe centralize all models/schemas à la Ruby on Rails?
Edit: I've been thinking about this myself now, and this might be more related to SQLAlchemy than Flask because you have to have the declarative_base() when declaring models. And that's got to come from somewhere, anyway!
Perhaps the best solution is to have your project's schema defined in one place and spread it around, like Ruby on Rails does. Declarative SQLAlchemy class definitions are really more like schema.rb than Django's models.py. I imagine this would also make it easier to use migrations (from alembic or sqlalchemy-migrate).
I was asked to provide an example, so let's do something simple: Say I have a blueprint describing "flatpages" -- simple, "static" content stored in the database. It uses a table with just shortname (for URLs), a title and a body. This is simple_pages/__init__.py:
from flask import Blueprint, render_template
from .models import Page
flat_pages = Blueprint('flat_pages', __name__, template_folder='templates')
#flat_pages.route('/<page>')
def show(page):
page_object = Page.query.filter_by(name=page).first()
return render_template('pages/{}.html'.format(page), page=page_object)
Then, it would be nice to let this blueprint define its own model (this in simple_page/models.py):
# TODO Somehow get ahold of a `db` instance without referencing the app
# I might get used in!
class Page(db.Model):
name = db.Column(db.String(255), primary_key=True)
title = db.Column(db.String(255))
content = db.Column(db.String(255))
def __init__(self, name, title, content):
self.name = name
self.title = title
self.content = content
This question is related to:
Flask-SQLAlchemy import/context issue
What's your folder layout for a Flask app divided in modules?
And various others, but all replies seem to rely on import the app's db instance, or doing the reverse. The "Large app how to" wiki page also uses the "import your app in your blueprint" pattern.
* Since the official documentation shows how to create routes, views, templates and assets in a Blueprint without caring about what app it's "in", I've assumed that Blueprints should, in general, be reusable across apps. However, this modularity doesn't seem that useful without also having independent models.
Since Blueprints can be hooked into an app more than once, it might simply be the wrong approach to have models in Blueprints?

I believe the truest answer is that modular blueprints shouldn't concern themselves directly with data access, but instead rely on the application providing a compatible implementation.
So given your example blueprint.
from flask import current_app, Blueprint, render_template
flat_pages = Blueprint('flat_pages', __name__, template_folder='templates')
#flat_pages.record
def record(state):
db = state.app.config.get("flat_pages.db")
if db is None:
raise Exception("This blueprint expects you to provide "
"database access through flat_pages.db")
#flat_pages.route('/<page>')
def show(page):
db = current_app.config["flat_pages.db"]
page_object = db.find_page_by_name(page)
return render_template('pages/{}.html'.format(page), page=page_object)
From this, there is nothing preventing you from providing a default implementation.
def setup_default_flat_pages_db(db):
class Page(db.Model):
name = db.Column(db.String(255), primary_key=True)
title = db.Column(db.String(255))
content = db.Column(db.String(255))
def __init__(self, name, title, content):
self.name = name
self.title = title
self.content = content
class FlatPagesDBO(object):
def find_page_by_name(self, name):
return Page.query.filter_by(name=name).first()
return FlatPagesDBO()
And in your configuration.
app.config["flat_pages.db"] = setup_default_flat_pages_db(db)
The above could be made cleaner by not relying in direct inheritance from db.Model and instead just use a vanilla declarative_base from sqlalchemy, but this should represent the gist of it.

I have similar needs of making Blueprints completely modular and having no reference to the App. I came up with a possibly clean solution but I'm not sure how correct it is and what its limitations are.
The idea is to create a separate db object (db = SQLAlchemy()) inside the blueprint and call the init_app() and create_all() methods from where the root app is created.
Here's some sample code to show how the project is structured:
The app is called jobs and the blueprint is called status and it is stored inside the blueprints folder.
blueprints.status.models.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy() # <--- The db object belonging to the blueprint
class Status(db.Model):
__tablename__ = 'status'
id = db.Column(db.Integer, primary_key=True)
job_id = db.Column(db.Integer)
status = db.Column(db.String(120))
models.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy() # <--- The db object belonging to the root app
class Job(db.Model):
__tablename__ = 'job'
id = db.Column(db.Integer, primary_key=True)
state = db.Column(db.String(120)
factory.py
from .blueprints.status.models import db as status_db # blueprint db
from .blueprints.status.routes import status_handler # blueprint handler
from .models import db as root_db # root db
from flask import Flask
def create_app():
app = Flask(__name__)
# Create database resources.
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////path/to/app.db'
root_db.init_app(app)
status_db.init_app(app) # <--- Init blueprint db object.
with app.app_context():
root_db.create_all()
status_db.create_all() # <--- Create blueprint db.
# Register blueprint routes.
app.register_blueprint(status_handler, url_prefix="/status")
return app
I tested it with gunicorn with gevent worker and it works. I asked a separate question about the robustness of the solution here:
Create one SQLAlchemy instance per blueprint and call create_all multiple times

You asked "Are Blueprints not meant to be independent of the app and be redistributable (à la Django apps)? "
The answer is yes. Blueprints are not similar to Django App.
If you want to use different app/configurations, then you need to use "Application Dispatching" and not blueprints. Read this
[1]: http://flask.pocoo.org/docs/patterns/appdispatch/#app-dispatch [1]
Also, the link here [1] http://flask.pocoo.org/docs/blueprints/#the-concept-of-blueprints [1]
It clearly says and I quote "A blueprint in Flask is not a pluggable app because it is not actually an application – it’s a set of operations which can be registered on an application, even multiple times. Why not have multiple application objects? You can do that (see Application Dispatching), but your applications will have separate configs and will be managed at the WSGI layer."