Develop Reference

FastAPI issues with MongoDB - TypeError: 'ObjectId' object is not iterable

I am having some issues inserting into MongoDB via FastAPI.
The below code works as expected. Notice how the response variable has not been used in response_to_mongo().
The model is an sklearn ElasticNet model.
app = FastAPI()
def response_to_mongo(r: dict):
client = pymongo.MongoClient("mongodb://mongo:27017")
db = client["models"]
model_collection = db["example-model"]
model_collection.insert_one(r)
#app.post("/predict")
async def predict_model(features: List[float]):
prediction = model.predict(
pd.DataFrame(
[features],
columns=model.feature_names_in_,
)
)
response = {"predictions": prediction.tolist()}
response_to_mongo(
{"predictions": prediction.tolist()},
)
return response
However when I write predict_model() like this and pass the response variable to response_to_mongo():
#app.post("/predict")
async def predict_model(features: List[float]):
prediction = model.predict(
pd.DataFrame(
[features],
columns=model.feature_names_in_,
)
)
response = {"predictions": prediction.tolist()}
response_to_mongo(
response,
)
return response
I get an error stating that:
TypeError: 'ObjectId' object is not iterable
From my reading, it seems that this is due to BSON/JSON issues between FastAPI and Mongo. However, why does it work in the first case when I do not use a variable? Is this due to the asynchronous nature of FastAPI?

As per the documentation:
When a document is inserted a special key, "_id", is automatically
added if the document doesn’t already contain an "_id" key. The value
of "_id" must be unique across the collection. insert_one() returns an
instance of InsertOneResult. For more information on "_id", see the
documentation on _id.
Thus, in the second case of the example you provided, when you pass the dictionary to the insert_one() function, Pymongo will add to your dictionary the unique identifier (i.e., ObjectId) necessary to retrieve the data from the database; and hence, when returning the response from the endpoint, the ObjectId fails getting serialized—since, as described in this answer in detail, FastAPI, by default, will automatically convert that return value into JSON-compatible data using the jsonable_encoder (to ensure that objects that are not serializable are converted to a str), and then return a JSONResponse, which uses the standard json library to serialise the data.
Solution 1
Use the approach demonstrated here, by having the ObjectId converted to str by default, and hence, you can return the response as usual inside your endpoint.
# place these at the top of your .py file
import pydantic
from bson import ObjectId
pydantic.json.ENCODERS_BY_TYPE[ObjectId]=str
return response # as usual
Solution 2
Dump the loaded BSON to valid JSON string and then reload it as dict, as described here and here.
from bson import json_util
import json
response = json.loads(json_util.dumps(response))
return response
Solution 3
Define a custom JSONEncoder, as described here, to convert the ObjectId into str:
import json
from bson import ObjectId
class JSONEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, ObjectId):
return str(o)
return json.JSONEncoder.default(self, o)
response = JSONEncoder().encode(response)
return response
Solution 4
You can have a separate output model without the 'ObjectId' (_id) field, as described in the documentation. You can declare the model used for the response with the parameter response_model in the decorator of your endpoint. Example:
from pydantic import BaseModel
class ResponseBody(BaseModel):
name: str
age: int
#app.get('/', response_model=ResponseBody)
def main():
# response sample
response = {'_id': ObjectId('53ad61aa06998f07cee687c3'), 'name': 'John', 'age': '25'}
return response
Solution 5
Remove the "_id" entry from the response dictionary before returning it (see here on how to remove a key from a dict):
response.pop('_id', None)
return response

POST/PUT input to Rest API

I am launching REST API build in python.
I am inputting a list as input and getting the required data.
example:
your.api.com/birth?name=James&date=2015-02-01&name=Robert&date=2020-01-01
from flask import request
#app.route('/birth')
def birth():
names = request.form.getlist('name')
dates = request.form.getlist('date')
As the number of inputs I have is huge, the end point URL is becoming huge. Is there any way to do the same using PUT or POST where I dump a doc in some format (say json) as my input?

Sure, put the request fields in a JSON and go through the dictionary
#app.route('/birth', methods=['POST'])
def birth():
data = request.get_json()
names = data["names"]
# etc.

When the input grows it is suggested to use a POST call rather than GET call.
from flask import request
#app.route('/birth')
def birth():
names = request.form.getlist('name')
dates = request.form.getlist('date')
Convert /birth to the following
from flask import request
#app.route('/birth', methods=['POST'])
def birth():
input = request.get_json()
# <Do the processing>
And in the POST call from client-side use a JSON like the following
{
name: [<array of values>],
date: [<array of values>]
}

How to take in JSON as input to python tornado "post" method

I am trying to use tornado to do a simple get and post method. Quite new to tornado framework. For the post I would like to take in a json as input, use that input to feed into another function that I have to execute another part of code. However I can't get tornado post method to work even with a simple self.write().
For my get method I am reading from an SQL database to get the status of a sensor and write that in a json format. The get method works perfectly! When I go to localhost:port# it reads out my get json file. For my post method I would like to take in a simple json of just one key:value which is a float number. I want to take that float number that the user specified in the json and use it in my flowMKS.set() function that will change the setpoint parameter of the sensor. I am not sure how to input a json into the post method and read it into a variable. I have some #commented code below that I tried and didn't work. However I went back to the basics and just did a self.write("Hello World") to see if the post was working. I can't get self.write to work either. Keep getting a 500 error message when i go to localhost:port#/flow_post. The variable flow_status was used in my get method.
The intended result would be to take in a json {"setpoint":45.5} into the post method. Use the number and insert into my flowMKS method to change a parameter on the sensor.
How would you take in a json to a post method and take the number from the json input and store in a variable?
class Current(tornado.web.RequestHandler):
def get(self):
global flow_status
time = flow_status[0]
ip = flow_status[1]
rate = flow_status[2]
setp = flow_status[3]
tempc = flow_status[4]
status = {"flow_controller":{
"time":time,
"ip":ip,
"rate_sccm":rate,
"setpoint":setp,
"temperature_c":tempc,
}
}
self.write(status)
class Update(tornado.web.RequestHandler):
# def prepare(self):
# if self.request.haders["Content-Type"].startswith("application/json"):
# self.json_args = json.loads(self.request.body)
# else:
# self.json_args = None
def post(self):
# #expecting body data to contain JSON so we use json.loads to decrypt the JSON into a dict
# data = json.loads(self.request.body)
#
# #Getting what the setpoint should be
# setpoint = self.json_args["setpoint"]
#
# #making the input a float
# setpoint = float(setpoint)
#
# #setting up connection with sensor
# flowMKS = FlowController(flow_status[1])
#
# #sending setpoint to sensor
# flowMKS.set(setpoint)
self.write("Hello World")
if __name__ == '__main__':
# global flow_status
#Below is creating the Tornado based API for get and post methods
tornado.options.parse_command_line()
app = tornado.web.Application(
handlers=[(r'/',Current), (r'/flow_post', Update)])
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(options.port)
#using PeriodicCallback to get info from the SQL database every 500 ms
PeriodicCallback(get_sql_status,500).start()
#starting the entire Tornado IOLoop
tornado.ioloop.IOLoop.current().start()

For uploading a file using Tornado you can use this function tornado.httputil.parse_body_arguments which will split the uploaded file content in a dictionary file_dict and other arguments in the FormData in the args_dict.
Sample code:
import tornado.httputil
import tornado.web
import tornado.escape
import json
import os
import sys
import traceback
class FileHandler(tornado.web.RequestHandler):
def _return_response(self, request, message_to_be_returned: dict, status_code):
"""
Returns formatted response back to client
"""
try:
request.set_header("Content-Type", "application/json; charset=UTF-8")
request.set_status(status_code)
#If dictionary is not empty then write the dictionary directly into
if(bool(message_to_be_returned)):
request.write(message_to_be_returned)
request.finish()
except Exception:
raise
def set_default_headers(self, *args, **kwargs):
self.set_header('Content-Type','text/csv')
self.set_header("Access-Control-Allow-Origin", "*")
self.set_header("Access-Control-Allow-Headers", "x-requested-with")
self.set_header("Access-Control-Allow-Methods", "*")
def post(self):
"""
This function reads an uploaded file
"""
try:
arg_dict = {}
file_dict = {}
tornado.httputil.parse_body_arguments(self.request.headers["Content-Type"], self.request.body, arg_dict, file_dict)
uploaded_file = file_dict['TestFile'][0]
if not uploaded_file:
return self._return_response(self, { 'message': 'No test file uploaded, please upload a test file' }, 400)
# File contents here
file_contents = str(uploaded_file['body'], "utf-8")
self.set_status(200)
self.finish()
except Exception as ex:
return self._return_response(self, { "message": 'Could not complete the request because of some error at the server!', "cause": ex.args[0], "stack_trace": traceback.format_exc(sys.exc_info()) }, 500)
You can alternatively use tornado.escape.json_decode to deserialize the request body into a dictionary and do something with it.
Sample code:
import tornado.gen
import tornado.web
import tornado.escape
import json
import os
import sys
import traceback
class JSONHandler(tornado.web.RequestHandler):
def _return_response(self, request, message_to_be_returned: dict, status_code):
"""
Returns formatted response back to client
"""
try:
request.set_header("Content-Type", "application/json; charset=UTF-8")
request.set_status(status_code)
#If dictionary is not empty then write the dictionary directly into
if(bool(message_to_be_returned)):
request.write(message_to_be_returned)
request.finish()
except Exception:
raise
def set_default_headers(self, *args, **kwargs):
self.set_header("Content-Type", "application/json")
self.set_header("Access-Control-Allow-Origin", "*")
self.set_header("Access-Control-Allow-Headers", "x-requested-with")
self.set_header("Access-Control-Allow-Methods", "*")
def post(self):
"""
This function parses the request body and does something
"""
try:
# Do something with request body
request_payload = tornado.escape.json_decode(self.request.body)
return self._return_response(self, request_payload, 200)
except json.decoder.JSONDecodeError:
return self._return_response(self, { "message": 'Cannot decode request body!' }, 400)
except Exception as ex:
return self._return_response(self, { "message": 'Could not complete the request because of some error at the server!', "cause": ex.args[0], "stack_trace": traceback.format_exc(sys.exc_info()) }, 500)

How to test render_to_response in Pyramid Python

I have a piece of code that calls Pyramid's render_to_response. I am not quite sure how to test that piece. In my test, the request object that is sent in is a DummyRequest by Pyramid. How can I capture to_be_rendered.
from pyramid.renderers import render_to_response
def custom_adapter(response):
data = {
'message': response.message
}
to_be_rendered = render_to_response(response.renderer, data)
to_be_rendered.status_int = response.status_code
return to_be_rendered

I believe render_to_response should return a response object. You should be able to call custom_adapter directly in your unit test, providing a DummyRequest and make assertions on the Response object returned by your custom_adapter
def test_custom_adapter(self):
dummy = DummyRequest() # not sure of the object here
response = custom_adapter(dummy)
self.assertEqual(response.status, 200)

Read data from HttpResponse using python script

I have one function in Python that returns json as HttpResponse for the client side as:-
def get_response():
return HttpResponse(json.dumps({'status':1}))
Now from the server itself I want a script to decode the value returned by the function itself and check the status value as:-
check_value = READ_THE_RESPONSE_VALUE(get_response())['status']
How can I do this??

You can do it this way:
import json
response = get_response()
json_response = json.loads(response.content)
json_response['status']