Develop Reference

Testing a POST method unit test which inserts data to mongodb database

I want to know how am I supposed to test my code and see whether it works properly. I want to make sure that it stores the received data to the database. Can you please tell me how am I supposed to do that? While I was searching the forum I found this post but I did not really understand what is going on. here is the code I want to test.
client = MongoClient(os.environ.get("MONGODB_URI"))
app.db = client.securify
app.secret_key = str(os.environ.get("APP_SECRET"))
#app.route("/", methods=["GET", "POST"])
def home():
if request.method == "POST":
ip_address = request.remote_addr
entry_content = request.form.get("content")
formatted_date = datetime.datetime.today().strftime("%Y-%m-%d/%H:%M")
app.db.entries.insert({"content": entry_content, "date": formatted_date, "IP": ip_address})
return render_template("home.html")
and here is the mock test I wrote:
import os
from unittest import TestCase
from app import app
class AppTest(TestCase):
# executed prior to each test
def setUp(self):
# you can change your application configuration
app.config['TESTING'] = True
# you can recover a "test cient" of your defined application
self.app = app.test_client()
# then in your test method you can use self.app.[get, post, etc.] to make the request
def test_home(self):
url_path = '/'
response = self.app.get(url_path)
self.assertEqual(response.status_code, 200)
def test_post(self):
url_path = '/'
response = self.app.post(url_path,data={"content": "this is a test"})
self.assertEqual(response.status_code, 200)
The test_post gets stuck and after some seconds gives an error when reaches app.db.entries.insert({"content": entry_content, "date": formatted_date, "IP": ip_address}) part. Please tell me also how can I retrieve the saved data in order to make sure it is saved in the expected way

This is what I do using NodeJS, not tested at all in python but the idea is the same.
First of all, find a in-memory DB, there are options like pymongo-inmemory or mongomock
Then in your code you have to do the connection according to you environment (production/development/whatever)
Something like this:
env = os.environ.get("ENV")
if env == "TESTING":
# connect to mock db
elif env == "DEVELOMPENT":
# for example if you want to test against a real DB but not the production one
# then do the connection here
else:
# connect to production DB

I don't know if it is the proper way to do it but I found a solution. After creating a test client self.app = app.test_client() the db gets set to localhost:27017 so I changed it manually as follows and it worked:
self.app = app.test_client()
client = MongoClient(os.environ.get("MONGODB_URI"))

Having Trouble Making a RESTful API with Flask-RestX: "No operations defined in spec!" and "404"s

In summary, I have been following the flask restx tutorials to make an api, however none of my endpoints appear on the swagger page ("No operations defined in spec!") and I just get 404 whenever I call them
I created my api mainly following this https://flask-restx.readthedocs.io/en/latest/scaling.html
I'm using python 3.8.3 for reference.
A cut down example of what I'm doing is as follows.
My question in short is, what am I missing?
Currently drawing blank on why this doesn't work.
Directory Structure
project/
- __init__.py
- views/
- __init__.py
- test.py
manage.py
requirements.txt
File Contents
requirements.txt
Flask-RESTX==0.2.0
Flask-Script==2.0.6
manage.py
from flask_script import Manager
from project import app
manager = Manager(app)
if __name__ == '__main__':
manager.run()
project/init.py
from flask import Flask
from project.views import api
app = Flask(__name__)
api.init_app(app)
project/views/init.py
from flask_restx import Api, Namespace, fields
api = Api(
title='TEST API',
version='1.0',
description='Testing Flask-RestX API.'
)
# Namespaces
ns_test = Namespace('test', description='a test namespace')
# Models
custom_greeting_model = ns_test.model('Custom', {
'greeting': fields.String(required=True),
})
# Add namespaces
api.add_namespace(ns_test)
project/views/test.py
from flask_restx import Resource
from project.views import ns_test, custom_greeting_model
custom_greetings = list()
#ns_test.route('/')
class Hello(Resource):
#ns_test.doc('say_hello')
def get(self):
return 'hello', 200
#ns_test.route('/custom')
class Custom(Resource):
#ns_test.doc('custom_hello')
#ns_test.expect(custom_greeting_model)
#ns_test.marshal_with(custom_greeting_model)
def post(self, **kwargs):
custom_greetings.append(greeting)
pos = len(custom_greetings) - 1
return [{'id': pos, 'greeting': greeting}], 200
How I'm Testing & What I Expect
So going to the swagger page, I expect the 2 endpoints defined to be there, but I just see the aforementioned error.
Just using Ipython in a shell, I've tried to following calls using requests and just get back 404s.
import json
import requests as r
base_url = 'http://127.0.0.1:5000/'
response = r.get(base_url + 'api/test')
response
response = r.get(base_url + 'api/test/')
response
data = json.dumps({'greeting': 'hi'})
response = r.post(base_url + 'test/custom', data=data)
response
data = json.dumps({'greeting': 'hi'})
response = r.post(base_url + 'test/custom/', data=data)
response

TL;DR
I made a few mistakes in my code and test:
Registering api before declaring the routes.
Making a wierd assumption about how the arguments would be passed to the post method.
Using a model instead of request parser in the expect decorator
Calling the endpoints in my testing with an erroneous api/ prefix.
In Full
I believe it's because I registered the namespace on the api before declaring any routes.
My understanding is when the api is registered on the app, the swagger documentation and routes on the app are setup at that point. Thus any routes defined on the api after this are not recognised. I think this because when I declared the namespace in the views/test.py file (also the model to avoid circular referencing between this file and views/__init__.py), the swagger documentation had the routes defined and my tests worked (after I corrected them).
There were some more mistakes in my app and my tests, which were
Further Mistake 1
In my app, in the views/test.py file, I made a silly assumption that a variable would be made of the expected parameter (that I would just magically have greeting as some non-local variable). Looking at the documentation, I learnt about the RequestParser, and that I needed to declare one like so
from flask_restx import reqparse
# Parser
custom_greeting_parser = reqparse.RequestParser()
custom_greeting_parser.add_argument('greeting', required=True, location='json')
and use this in the expect decorator. I could then retrieve a dictionary of the parameters in my post method. with the below
...
def post(self):
args = custom_greeting_parser.parse_args()
greeting = args['greeting']
...
The **kwargs turned out to be unnecessary.
Further Mistake 2
In my tests, I was calling the endpoint api/test, which was incorrect, it was just test. The corrected test for this endpoint is
Corrected test for test endpoint
import json
import requests as r
base_url = 'http://127.0.0.1:5000/'
response = r.get(base_url + 'test')
print(response)
print(json.loads(response.content.decode()))
Further Mistake 3
The test for the other endpoint, the post, I needed to include a header declaring the content type so that the parser would "see" the parameters, because I had specified the location explictily as json. Corrected test below.
Corrected test for test/custom endpoint
import json
import requests as r
base_url = 'http://127.0.0.1:5000/'
data = json.dumps({'greeting': 'hi'})
headers = {'content-type': 'application/json'}
response = r.post(base_url + 'test/custom', data=data, headers=headers)
print(response)
print(json.loads(response.content.decode()))
Corrected Code
For the files with incorrect code.
views/init.py
from flask_restx import Api
from project.views.test import ns_test
api = Api(
title='TEST API',
version='1.0',
description='Testing Flask-RestX API.'
)
# Add namespaces
api.add_namespace(ns_test)
views/test.py
from flask_restx import Resource, Namespace, fields, reqparse
# Namespace
ns_test = Namespace('test', description='a test namespace')
# Models
custom_greeting_model = ns_test.model('Custom', {
'greeting': fields.String(required=True),
'id': fields.Integer(required=True),
})
# Parser
custom_greeting_parser = reqparse.RequestParser()
custom_greeting_parser.add_argument('greeting', required=True, location='json')
custom_greetings = list()
#ns_test.route('/')
class Hello(Resource):
#ns_test.doc('say_hello')
def get(self):
return 'hello', 200
#ns_test.route('/custom')
class Custom(Resource):
#ns_test.doc('custom_hello')
#ns_test.expect(custom_greeting_parser)
#ns_test.marshal_with(custom_greeting_model)
def post(self):
args = custom_greeting_parser.parse_args()
greeting = args['greeting']
custom_greetings.append(greeting)
pos = len(custom_greetings) - 1
return [{'id': pos, 'greeting': greeting}], 200

connecting mongodb server via seperate class

I am using flask to create simple api. The api simply returns values from mongoDB. Everything works great if i do the connection within same function. I am not doing connection simply at start of file because i am using uwsgi and nginx server on ubuntu. If i do that then there will be a problem of fork.
However, I have to use this connection with other api so thought to make a seperate class for connection and each api will simply call it . I m using this functionality to make codes manageable. However when i try the these codes it always shows internal server error. I tried making this function static too , still the error exists.
Note - I have replaced mongodb address with xxx as i am using mongodbatlas account here
from flask import Flask
from flask import request, jsonify
from flask_pymongo import pymongo
from pymongo import MongoClient
from flask_restful import Resource, Api, reqparse
app = Flask(__name__)
api = Api(app)
#client = MongoClient("xxx")
#db = client.get_database('restdb')
#records = db.stars
class dbConnect():
def connect(self):
client = MongoClient("xxx")
db = client.get_database('restdb')
records = db.stars
return records
class Order(Resource):
def get(self):
#client = MongoClient("xxx")
#db = client.get_database('restdb')
#records = db.stars
#star = records
star = dbConnect.connect
output = []
for s in star.find():
output.append({'name' : s['name'], 'distance' : s['distance']})
return jsonify({'result' : output})
api.add_resource(Order, '/')
if __name__ == "__main__":
app.run(host='0.0.0.0')
ERROR {"message": "Internal Server Error"}

Preliminary investigation suggests that you haven't instantiated your dbConnect class. Also, you haven't called the method connect properly.
class Order(Resource):
def get(self):
db = dbConnect() # This was missing
star = db.connect() # This is how you make method call properly.
output = []
for s in star.find():
output.append({'name' : s['name'], 'distance' : s['distance']})
return jsonify({'result' : output})
Also class dbConnect() should be declared as class dbConnect:.

Flask test_client removes query string parameters

I am using Flask to create a couple of very simple services. From outside testing (using HTTPie) parameters through querystring are getting to the service.
But if I am using something like.
data = {
'param1': 'somevalue1',
'param2': 'somevalue2'}
response = self.client.get(url_for("api.my-service", **data))
I can see the correct URI being created:
http://localhost:5000/api1.0/my-service?param1=somevalue1&param2=somevalue2
when I breakpoint into the service:
request.args
is actually empty.
self.client is created by calling app.test_client() on my configured Flask application.
Anyone has any idea why anything after ? is being thrown away or how to work around it while still using test_client?

I've just found out a workaround.
Make
data = {
'param1': 'somevalue1',
'param2': 'somevalue2'}
response = self.client.get(url_for("api.my-service", **data))
into this:
data = {
'param1': 'somevalue1',
'param2': 'somevalue2'}
response = self.client.get(url_for("api.my-service"), query_string = data)
This works but seems a bit unintuitive, and debugging there is a place where the provided query string in the URI is thrown away ....
But anyway this works for the moment.

I know this is an old post, but I ran into this too. There's an open issue about this in the flask github repository. It appears this is intended behavior. From a response in the issue thread:
mitsuhiko commented on Jul 24, 2013
That's currently intended behavior. The first parameter to the test client should be a relative url. If it's not, then the parameters are removed as it's treated as if it was url joined with the second. This works:
>>> from flask import Flask, request
>>> app = Flask(__name__)
>>> app.testing = True
>>> #app.route('/')
... def index():
... return request.url
...
>>> c = app.test_client()
>>> c.get('/?foo=bar').data
'http://localhost/?foo=bar'
One way to convert your absolute url into a relative url and keep the query string is to use urlparse:
from urlparse import urlparse
absolute_url = "http://someurl.com/path/to/endpoint?q=test"
parsed = urlparse(absolute_url)
path = parsed[2] or "/"
query = parsed[4]
relative_url = "{}?{}".format(path, query)
print relative_url

If you are trying any other HTTP Method other than GET
response = self.client.patch(url_for("api.my-service"), query_string=data,
data="{}")
data="{}" or data=json.dumps({}) should be there, even if there is no content in the body. Otherwise, it results in BAD Request.

For me the solution was to use the client within with statements:
with app.app_context():
with app.test_request_context():
with app.test_client() as client:
client.get(...)
instead of
client = app.test_client()
client.get(...)
I put the creation of the test client in a fixture, so that it is "automatically" created for each test method:
from my_back_end import MyBackEnd
sut = None
app = None
client = None
#pytest.fixture(autouse=True)
def before_each():
global sut, app, client
sut = MyBackEnd()
app = sut.create_application('frontEndPathMock')
with app.app_context():
with app.test_request_context():
with app.test_client() as client:
yield

How to speed up JSON for a flask application?

I'm currently implementing a webapp in flask. It's an app that does a visualization of data gathered. Each page or section will always have a GET call and each call will return a JSON response which then will be processed into displayed data.
The current problem is that some calculation is needed before the function could return a JSON response. This causes some of the response to arrive slower than others and thus making the page loads a bit slow. How do I properly deal with this? I have read into caching in flask and wonder whether that is what the app need right now. I have also researched a bit into implementing a Redis-Queue. I'm not really sure which is the correct method.
Any help or insights would be appreciated. Thanks in advance

Here are some ideas:
If the source data that you use for your calculations is not likely to change often then you can run the calculations once and save the results. Then you can serve the results directly for as long as the source data remains the same.
You can save the results back to your database, or as you suggest, you can save them in a faster storage such as Redis. Based on your description I suspect the big performance gain will be in not doing calculations so often, the difference between storing in a regular database vs. Redis or similar is probably not significant in comparison.
If the data changes often then you will still need to do calculations frequently. For such a case an option that you have is to push the calculations to the client. Your Flask app can just return the source data in JSON format and then the browser can do the processing on the user's computer.
I hope this helps.

You can use
copy_current_request_context and Redis, Thread
It is helpful when you need long time to make JSON response.
The first request maybe slow, but next request will faster.
Example
from datetime import timedelta, datetime
from threading import Thread
from . import dbb, redis_client
from flask import Blueprint, request, jsonify, flash, after_this_request, copy_current_request_context, \
current_app, send_from_directory
from .models import Shop, Customers
def save_customer_json_to_redis(request):
response_json = {
"have_customer": False,
"status": False,
"anythingelse": None,
"message":"False, you have to check..."
}
#print(request.data)
headers = request.headers
Authorization = headers['Authorization']
token = Authorization.replace("Bearer", "")
phone = request.args.get('phone')
if phone is not None and phone != "":
print('token', token, "phone", phone)
now = datetime.utcnow() + timedelta(hours=7)
shop = Shop.query.filter(Shop.private_token == token, Shop.ended_date > now, Shop.shop_active == True).first()
customer = Customers.query.filter_by(shop_id=shop.id, phone=phone).first()
if customer:
redis_name = f'{shop.id}_api_v2_customer_phone_{phone}_customer_id_{customer.id}'
print(redis_name)
response_json["anythingelse"] = ...# do want you want, it need long time to do
response_json["status"] = True
response_json["message"] = "Successful"
redis_client.set(redis_name, json.dumps(response_json)) #Save JSON to Redis
#app.route('/api/v2/customer', methods=['GET'])
def api_customer():
#copy_current_request_context
def do_update_customer_to_redis():# this function to save JSON you want to response next time to Redis
save_customer_json_to_redis(request)
Thread(target=do_update_customer_to_redis).start()
response_json = {
"have_customer": False,
"status": False,
"anythingelse": {},
"message": "False, you have to check..."
}
#print(request.data)
headers = request.headers
Authorization = headers['Authorization']
token = Authorization.replace("Bearer", "")
phone = request.args.get('phone')
if phone is not None and phone != "":
print('token', token, "phone", phone)
now = datetime.utcnow() + timedelta(hours=7)
shop = Shop.query.filter(Shop.private_token == token, Shop.ended_date > now,Shop.shop_active == True).first()
customer = Customers.query.filter_by(shop_id=shop.id, phone=phone).first()
if customer:
redis_name = f'{shop.id}_api_v2_customer_phone_{phone}_customer_id_{customer.id}'
print(redis_name)
try:
response_json = json.loads(redis_client.get(redis_name)) # if have json from app
print("json.loads(redis_client.get(redis_name))")
except Exception as e:
print("json.loads(redis_client.get(redis_name))", e)
#do any thing you want to response json
response_json["anythingelse"] = ...# do want you want, it need long time to do
response_json["message"]= ...#do want you want
#redis_client.set(redis_name, json.dumps(response_json))
response_json["status"] = True
response_json["message"] = "Successful"
return jsonify(response_json)
In the init.py
from flask import Flask
from flask_cors import CORS
from flask_mail import Mail
from flask_sqlalchemy import SQLAlchemy
from redis import Redis
# init SQLAlchemy so we can use it later in our models
dbb = SQLAlchemy(session_options={"autoflush": False})
redis_client = Redis(
host='localhost',
port='6379',
password='your_redis_password'
)
def create_app():
app = Flask(__name__)
...........