I am currently developing an application with Flask-Restless. When I substituted my SQLAlchemy models' typical DateTime fields with corresponding arrow fields, all went smoothly. This was due to the help of SQLAlchemy-Utils and its ArrowType field.
However, after using the API to return a JSON representation of these objects, I received the following error:
TypeError: Arrow [2015-01-05T01:17:48.074707] is not JSON serializable
Where would be the ideal place to modify how the model gets serialized? Do I modify Flask-Restless code to support Arrow objects or write a model method that Flask-Restless can identify and use to retrieve a JSON-compatible object?
I could also write an ugly post-processor function in the meantime but that solution seems like a terrible hack.
Below is an example model with the ArrowType field:
class Novel(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.Unicode, unique=True, nullable=False)
created_at = db.Column(ArrowType, nullable=False)
def __init__(self, title):
self.title = title
self.created_at = arrow.utcnow()
Arrow now has a for_json method. For example: arrow.utcnow().for_json()
How about a custom JSONEncoder which supports Arrow types? Looking at the Flask-Restless source code, it uses Flask's built in jsonify under the hood. See this snippet for an example which serializes regular datetime objects in a different format: http://flask.pocoo.org/snippets/119/
Here's a full self-contained example for good measure:
import flask
import flask.ext.sqlalchemy
import flask.ext.restless
from flask.json import JSONEncoder
import sqlalchemy_utils
import arrow
app = flask.Flask(__name__)
app.config['DEBUG'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db = flask.ext.sqlalchemy.SQLAlchemy(app)
class Event(db.Model):
id = db.Column(db.Integer, primary_key=True)
timestamp = db.Column(sqlalchemy_utils.ArrowType)
class ArrowJSONEncoder(JSONEncoder):
def default(self, obj):
try:
if isinstance(obj, arrow.Arrow):
return obj.format('YYYY-MM-DD HH:mm:ss ZZ')
iterable = iter(obj)
except TypeError:
pass
else:
return list(iterable)
return JSONEncoder.default(self, obj)
app.json_encoder = ArrowJSONEncoder
db.create_all()
manager = flask.ext.restless.APIManager(app, flask_sqlalchemy_db=db)
manager.create_api(Event, methods=['GET','POST'])
app.run()
From the two options in your post, I'd suggest adding the method to your model class to retrieve a JSON-compatible object, only because it's simpler and more maintainable. If you want to modify Flask-Restless, you either need to fork it or monkey patch it.
Related
This question already has answers here:
FastAPI - GET request results in typeerror (value is not a valid dict)
(2 answers)
Closed 10 months ago.
I started to build a wepapp with fastapi and Vue. I started at the backend and tried to get some data from a database. As ORM I use SQLAlchemy. What I am tring to achive is to strip down the DB response to some specific columns.
I am fairly new to the world of webapplications, I will also appreciate some good resources. I find it kind of hard to get started in this topic. Despite there being alt of Tutorials, they just cover how to start a basic site and leave the rest to the docs. But I you are not used to the terminology, its quite easy to get lost there.
Anyway, I have this test setup:
My model for the db is
class System(Base):
__tablename__ = 'system'
id = Column(BIGINT(20), primary_key=True)
name = Column(String(200), nullable=False)
type = Column(String(200), nullable=False)
installed_power = Column(Float(asdecimal=True), nullable=False)
date_of_installation = Column(
DateTime, nullable=False, server_default=text("current_timestamp()"))
last_changed = Column(DateTime, nullable=False, server_default=text(
"current_timestamp() ON UPDATE current_timestamp()"))
site_id = Column(ForeignKey('site.id'), nullable=False, index=True)
site = relationship('Site')
and my schema is
class System(BaseModel):
id: int
name: str
type: str
installed_power: int
In my main.py I am doing this
#app.get("/system", response_model=schemas.System)
def get_systems(db: Session = Depends(get_db)):
query = crud.get_system(db)
return query.all()
This dose not work. The error says pydantic.error_wrappers.ValidationError: 1 validation error for System response value is not a valid dict (type=type_error.dict)
If I add all db columns to the schema it works obviously
I also tried something like this, but this did not work either.
#app.get("/system", response_model=schemas.System)
def get_systems(db: Session = Depends(get_db)):
query = crud.get_system(db)
return query.all()
res = []
for row in query:
system = schemas.System()
system.id = row.id
system.name = row.name
system.type = row.type
system.installed_power = row.installed_power
res.append(system)
return res
WITHOUT orm_mode
In your first example, your response_model expects a schema (System), but you send it query.all() which returns a list containing classes from the model you get. (SQLAlchemy does not return a dictionary, which is what pydantic expects by default).
So, at this point, you have to make a choice, either your endpoint must return ONE object, in which case you should not use query.all() but something like query.one_or_none().
Or you want to return a list of objects and your response_model should be:
from typing import List
#app.get("/system", response_model=List[schemas.System])
All that remains is to format your data so that it corresponds to your schema.
for version with only one data:
#app.get("/system", response_model=schemas.System)
def get_systems(db: Session = Depends(get_db)):
query = crud.get_system(db).one_or_none()
return schemas.System(id=query.id,name= query.name, type=query.type, installed_power=query.installed_power)
for version with multiple datas:
from typing import List
def make_response_systems(query):
result = []
for data in query:
result.append(schemas.System(id=query.id,name= query.name, type=query.type, installed_power=query.installed_power))
return result
#app.get("/system", response_model=List[schemas.System])
def get_systems(db: Session = Depends(get_db)):
query = crud.get_system(db).all()
return make_response_systems(query)
There are more aesthetic ways to do this. But I think the example above is a good way to understand how it works.
For the rest, you can look at the orm_mode of pydantics models, which you can find in the FastAPI documentation.
To learn, the FastAPI documentation is very complete and easy to access.
WITH orm_mode
Pydantic's orm_mode will tell the Pydantic model to read the data even if it is not a dict, but an ORM model (or any other arbitrary object with attributes).
class System(BaseModel):
id: int
name: str
type: str
installed_power: int
class Config:
orm_mode = True
#app.get("/system", response_model=schemas.System)
def get_systems(db: Session = Depends(get_db)):
query = crud.get_system(db).one_or_none()
return query
Documentation fastapi orm_mode : https://fastapi.tiangolo.com/tutorial/sql-databases/?h=orm_mode#use-pydantics-orm_mode
As per pydantic docs
ORM Mode (aka Arbitrary Class Instances)🔗
Pydantic models can be created from arbitrary class instances to support models that map to ORM objects.
To do this:
The Config property orm_mode must be set to True.
The special constructor from_orm must be used to create the model instance.
we need to add orm_mode to the schema config.
class System(BaseModel):
id: int
name: str
type: str
installed_power: int
class Config:
orm_mode = True
Reference: https://pydantic-docs.helpmanual.io/usage/models/#orm-mode-aka-arbitrary-class-instances
I was making a sample Fast Api server with Tortoise ORM as an asynchronous orm library, but I just cannot seem to return the relations I have defined. These are my relations:
# Category
from tortoise.fields.data import DatetimeField
from tortoise.models import Model
from tortoise.fields import UUIDField, CharField
from tortoise.fields.relational import ManyToManyField
from tortoise.contrib.pydantic import pydantic_model_creator
class Category(Model):
id = UUIDField(pk=True)
name = CharField(max_length=255)
description = CharField(max_length=255)
keywords = ManyToManyField(
"models.Keyword", related_name="categories", through="category_keywords"
)
created_on = DatetimeField(auto_now_add=True)
updated_on = DatetimeField(auto_now=True)
Category_dto = pydantic_model_creator(Category, name="Category", allow_cycles = True)
# Keyword
from models.expense import Expense
from models.category import Category
from tortoise.fields.data import DatetimeField
from tortoise.fields.relational import ManyToManyRelation
from tortoise.models import Model
from tortoise.fields import UUIDField, CharField
from tortoise.contrib.pydantic import pydantic_model_creator
class Keyword(Model):
id = UUIDField(pk=True)
name = CharField(max_length=255)
description = CharField(max_length=255)
categories: ManyToManyRelation[Category]
expenses: ManyToManyRelation[Expense]
created_on = DatetimeField(auto_now_add=True)
updated_on = DatetimeField(auto_now=True)
class Meta:
table="keyword"
Keyword_dto = pydantic_model_creator(Keyword)
The tables have been created correctly. When adding keywords to categories the db state is all good. The problem is when i want to query the categories and include the keywords. I have this code for that:
class CategoryRepository():
#staticmethod
async def get_one(id: str) -> Category:
category_orm = await Category.get_or_none(id=id).prefetch_related('keywords')
if (category_orm is None):
raise NotFoundHTTP('Category')
return category_orm
Debugging the category_orm here I have the following:
category_orm debug at run-time
Which kind of tells me that they are loaded.
Then when i cant a Pydantic model I have this code
class CategoryUseCases():
#staticmethod
async def get_one(id: str) -> Category_dto:
category_orm = await CategoryRepository.get_one(id)
category = await Category_dto.from_tortoise_orm(category_orm)
return category
and debugging this, there is no keywords field
category (pydantic) debug at run-time
Looking at the source code of tortoise orm for the function from_tortoise_orm
#classmethod
async def from_tortoise_orm(cls, obj: "Model") -> "PydanticModel":
"""
Returns a serializable pydantic model instance built from the provided model instance.
.. note::
This will prefetch all the relations automatically. It is probably what you want.
But my relation is just not returned. Anyone have a similar experience ?
The issue occurs when one try to generate pydantic models before Tortoise ORM is initialised. If you look at basic pydantic example you will see that all pydantic_model_creator are called after Tortoise.init.
The obvious solution is to create pydantic models after Tortoise initialisation, like so:
await Tortoise.init(db_url="sqlite://:memory:", modules={"models": ["__main__"]})
await Tortoise.generate_schemas()
Event_Pydantic = pydantic_model_creator(Event)
Or a more convenient way, use early model init by means of Tortoise.init_models(). Like so:
from tortoise import Tortoise
Tortoise.init_models(["__main__"], "models")
Tournament_Pydantic = pydantic_model_creator(Tournament)
In the case, the main idea is to split pydantic and db models into different modules, so that importing the first does not lead to the creation of the second ahead of time. And ensure calling Tortoise.init_models() before creating pydantic models.
A more detailed description with examples can be found here.
I'm using flask with flask_sqlalchemy and I'm a bit perplexed.
This code runs but when you run db.create_all() the database you get is empty with not tables.
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
db.create_all()
class Urls(db.Model):
id = db.Column(db.Integer, primary_key=True) # autoincrement=True)
name = db.Column(db.String(100), unique=True, nullable=False)
title = db.Column(db.String(100))
zone = db.Column(db.Integer, default=10, nullable=False)
The fix is to push the definition of the Users class above the db.create_all() line. Then you get a database with the users table inside it.
My Question is how can db.create_all() know that the Users class is now defined.
is create_all somehow importing the file again?
Furthermore how does it know to use Urls and not any other class.
This seems like black magic to me.
If you approach this by finding and reading the flask_sqlalchemy source, it might well appear to be black magic. That's not easy code to follow for someone new to Python. The sqlalchemy source is even deeper magic.
A somewhat simpler question is to answer is how a given class can locate its subclasses. For that, Python classes have a special __subclasses__ method that returns a list of (weak) references to immediate subclasses. With that and a bit of extra work, it's possible to walk a tree of subclasses.
For example, if Bar is a subclass of Foo:
>>> class Foo: pass
...
>>> class Bar(Foo): pass
...
>>> Foo.__subclasses__()
[<class '__main__.Bar'>]
See https://docs.python.org/3/library/stdtypes.html near the bottom.
I'm using flask-restless and getting a Bad Request (400) error on my first POST (from Postman). If I keep on making the same request, it keeps on erroring. But if I remove the field that flask-restless complains about, run the POST again, get a positive response, add that same field back in, and run it again, it works fine from then on.
URL: /api/appraisals
Request JSON:
{
"suggested_price": 88,
"listing": {"id": 1}
}
Error Response:
{
"message": "Model does not have field 'listing'"
}
app.py:
from models.db import init_app
from controllers import api
app = Flask(__name__)
app.config.from_object('config')
init_app(app)
api.init_api(app)
from models.db.py:
from flask.ext.sqlalchemy import SQLAlchemy
def init_app(app):
with app.app_context():
db.init_app(app)
db.create_all()
db = SQLAlchemy()
from controllers.api.py:
from flask.ext.restless import APIManager
class ResourceManager(APIManager): ...
def init_api(app):
with app.app_context():
manager = ResourceManager(app, flask_sqlalchemy_db=db)
manager.add_resource(ListingResource())
manager.add_resource(AppraisalResource())
from models.appraisal.py:
from .db import db
from .base import BaseModel
class Appraisal(BaseModel):
__tablename__ = "appraisal"
# required fields
suggested_price = db.Column(db.Integer, nullable=False)
# optional fields
currency = db.Column(db.Unicode, default=u"USD")
# relationships
listing_id = db.Column(db.Integer, db.ForeignKey('listing.id'))
from models.listing.py:
from sqlalchemy.schema import UniqueConstraint
from .db import db
from .base import BaseModel
class Listing(StatusfulModel, BaseModel):
__tablename__ = "listing"
# relationships
appraisals = db.relationship(
"Appraisal",
backref=db.backref("listing", uselist=False),
uselist=True)
from controllers.resource.appraisal.py:
class AppraisalResource(Resource):
model_class = Appraisal
base_url = "appraisals"
allowed_methods = ["POST", "GET"]
def get_fields(self):
super_fields = super(AppraisalResource, self).get_fields()
return super_fields + [
"listing",
"listing.id"
]
I think the best way to explain how to solve this error is to tell you what was on my mind while thinking about that.
You going to get into those situation a lot with APIs, and it's importing to know how to think about the logics.
You got error 400 - that mean you sent a JSON but not in the format Flask expecting.
According to your question removing the listing field solved the problem - feels like we getting closer.
I looked on what model you trying to make the change(/api/appraisals) - Appraisals
Your model:
class Appraisal(BaseModel):
__tablename__ = "appraisal"
suggested_price = db.Column(db.Integer, nullable=False)
currency = db.Column(db.Unicode, default=u"USD")
listing_id = db.Column(db.Integer, db.ForeignKey('listing.id'))
After looking at this, you can see Flask expecting "listing_id" - and not "listing"
I hope i was clear, tell me if you need any help.
Goodluck!
Currently, the way our, as well as most web frameworks', serialization works is there's some type of method invocation which dumps the model into some type of format. In our case, we have a to_dict() method on every model that constructs and returns a key-value dictionary with the key being the field name and the value being the instance variable.
All throughout our code, we have snippets like the following: json.dumps(**some_model_object.to_dict()) which will serialize a some_model_object to json. Recently, we've decided to expose some internal resources to our users, but some of these resources have specific private instance values that we do not want to transmit back during serialization if the requesting user is not a super user.
I'm trying to come up with a clean design that will allow easier serialization, as well as allow us to serialize to a format other than json. I think this is a pretty good use case for Aspect Oriented Design/Programming, where the aspects respect the requesting access controls and serialize the object based on the requesting user's persmissions.
Here's something similar to what I have now:
from framework import current_request
class User(SQLAlchemyDeclarativeModel):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
first_name = Column(Unicode(255))
last_name = Column(Unicode(255))
private_token = Column(Unicode(4096))
def to_dict(self):
serialized = dict((column_name, getattr(self, column_name))
for column_name in self.__table__.c.keys())
# current request might not be bound yet, could be in a unit test etc.
if current_request and not current_request.user.is_superuser():
# we explicitly define the allowed items because if we accidentally add
# a private variable to the User table, then it might be exposed.
allowed = ['id', 'first_name', 'last_name']
serialized = dict((k, v) for k, v in serialized.iteritems() if k in allowed)
return serialized
As one can see, this is less than ideal because now I have to couple the database model with the current request. While this is very explicit, the request coupling is a code smell and I'm trying to see how to do this cleanly.
One way I've thought about doing it is to register some fields on the model like so:
class User(SQLAlchemyDeclarativeModel):
__tablename__ = 'users'
__public__ = ['id', 'first_name', 'last_name']
__internal__ = User.__exposed__ + ['private_token']
id = Column(Integer, primary_key=True)
first_name = Column(Unicode(255))
last_name = Column(Unicode(255))
private_token = Column(Unicode(4096))
Then, I would have a serializer class that is bound with the current request on every WSGI call that will take the desired serializer. For example:
import simplejson
from framework import JSONSerializer # json serialization strategy
from framework import serializer
# assume response format was requested as json
serializer.register_serializer(JSONSerializer(simplejson.dumps))
serializer.bind(current_request)
Then in my view somewhere, I would just do:
from framework import Response
user = session.query(User).first()
return Response(code=200, serializer.serialize(user))
serialize would be implemented as follows:
def serialize(self, db_model_obj):
attributes = '__public__'
if self.current_request.user.is_superuser():
attributes = '__private__'
payload = dict((c, getattr(db_model_obj, c))
for c in getattr(db_model_obj, attributes))
return self.serialization_strategy.execute(payload)
Thoughts on this approach's readability and clarity? Is this a pythonic approach to the problem?
Thanks in advance.
establish the "serialization" contract via a mixin:
class Serializer(object):
__public__ = None
"Must be implemented by implementors"
__internal__ = None
"Must be implemented by implementors"
def to_serializable_dict(self):
# do stuff with __public__, __internal__
# ...
keep it simple with the WSGI integration. "register", JSONSerializer as an object, and all that is some kind of Java/Spring thing, don't need that fanfare. Below is my pylons 1.0-style solution, I'm not on pyramid yet:
def my_controller(self):
# ...
return to_response(request, response, myobject)
# elsewhere
def to_response(req, resp, obj):
# this would be more robust, look in
# req, resp, catch key errors, whatever.
# xxx_serialize are just functions. don't need state
serializer = {
'application/json':json_serialize,
'application/xml':xml_serialize,
# ...
}[req.headers['content-type']]
return serializer(obj)