I'm currently stuck with a serialization problem. The data seems ok in Nameko but not in Flask because an error occurred. I tried to serialise and unserialize, but it didn't work.
Here is an example of data from a MongoDB database :
{u'building': u'urk', u'idcapteur': 8, u'room': u'tyu', u'idpiece': 1, u'uri': u'urk/tyu/luminosity/yota1', u'idMesure': 458945, u'subId': u'yota1', u'datemesure2': datetime.datetime(2017, 6, 16, 12, 48, 19, 179000), u'datemesure': datetime.datetime(2017, 6, 16, 12, 48, 19, 179000), u'device': u'luminosity', u'_id': ObjectId('7825ahy'), u'data': {u'date': u'2017-06-16T14:48:19.179435', u'payload': {u'subID': u'yota1', u'input': 50, u'value_units': u'lux', u'value': 500, u'unitID': u'inside'}, u'uri': u'urk/tyu/luminosity/yota1'}, u'mesurevaleur': [{u'idlibv': 5, u'valeur': 500.0}]}
Here is my Nameko program :
# -*-coding:utf-8 -*
# neocampus.py
import json
from database import startMongoDbConnection, getRawDataCollection, mongoDbQuery
from nameko.rpc import rpc
#from database.py import startMongoDbConnection, getRawDataCollection, mongoDbQuery
mdbConnection = ''
mdbCollection = ''
# Send a request to the microservices system. It will call other microservices
class MsRequest:
name="msRequest"
# TODO
#rpc
def msRequest(self, value):
# Connect to MongoDB database and collection, if not already done
global mdbConnection, mdbCollection, queryResult
if mdbConnection == '':
mdbConnection = startMongoDbConnection()
if mdbCollection == '':
mdbCollection = getRawDataCollection(mdbConnection)
# Check the requested microservice
someData = GetData()
jsonData = someData.getData(value)
# It prints the proper data
for document in jsonData:
print(document)
print("FINISHED !!!")
# Don't forget : closeMongoDbConnection()
return jsonData
# Get data depending on sensor type
class GetData:
name="getData"
def getData(self, sensorType):
global mdbCollection
queryResult = mongoDbQuery(mdbCollection, "device", "$eq", sensorType)
return queryResult
# From database.py
def mongoDbQuery(someCollection, key, choice, value):
# Use MongoDB query dependeing on choices in parameters
docs = someCollection.find({key:{choice:value}}).limit(10)
#for document in docs:
#print(document)
return docs
# Start a MongoDB connection
def startMongoDbConnection():
connected = connectToMongoDb("mongodb")
return connected
# Start a MongoDB connection
def startMongoDbConnection():
connected = connectToMongoDb("mongodb")
return connected
# Choose the first collection available (for raw data)
def getRawDataCollection(connection):
currentCollection = chooseCollection(connection, mongollection)
return currentCollection
And now, my Flask program :
#app.route('/', methods=['GET', 'POST'])
def getData():
with ClusterRpcProxy(CONFIG) as rpc:
myData = rpc.msRequest.msRequest("luminosity")
#for document in myData:
#print myData
return flask.jsonify(**myData)
Finally, the error output :
ERROR in app: Exception on / [GET]
Traceback (most recent call last):
File "/home/oyo/.local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/home/oyo/.local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/oyo/.local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/oyo/.local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/home/oyo/.local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/oyo/flask_sandfox/sandfox.py", line 16, in getData
myData = rpc.msRequest.msRequest("luminosity")
File "/home/oyo/.local/lib/python2.7/site-packages/nameko/rpc.py", line 369, in __call__
return reply.result()
File "/home/oyo/.local/lib/python2.7/site-packages/nameko/rpc.py", line 327, in result
raise deserialize(error)
RemoteError: UnserializableValueError Unserializable value: `<pymongo.cursor.Cursor object at 0x7f3274612e50>`
With a print, data is correctly displayed in the terminal. But not in Flask, I get an error 500. I thought that with JSON data and Jsonify (found here, in this site), I could display something. But it's not the cas.
Any help, advice or hint would be gladly appreciated. Thank you.
For people who'll have this exact problem, it took me 5 days to find it. I was sure it was in JSON, because, when I copied/pasted the result, it was ok. In fact, it was in BSON (I didn't even know this format). So, just add this at the beginning of your file :
from bson.json_util import dumps
Then, add this :
# limit for the example
retrievedocs = someCollection.find({* something *}}, {'_id': 0}).limit(10)
docs = dumps(retrievedocs)
Then, the serialization between Nameko and Flask will work.
Related
I'm developing an application with python and Flask to automate sending emails.
When I run this application locally, there is no problem, but after deploying the application in Azure, this error appears
TypeError: Object of type NoSectionError is not JSON serializable
The function that is bursting the error is this, when I try to return the dictionary variable
#app.route('/send_email', methods=['POST'])
def get_parameters():
"""
It receives a JSON object, prints it, and then calls another function that sends an email
:return: The result of the function send_email_to_customer
"""
user_input = request.json
print("Conteudo ", user_input)
try:
user = get_config_data(iten_title='EMAIL_LOGIN', iten='email')
psw = get_config_data(iten_title='EMAIL_LOGIN', iten='password')
id_meeting = user_input['id']
list_of_recipients = user_input['recipients']
creator_name = user_input['creator_name']
date_list = user_input['meeting_day']
subject = f'Melhor data para a reuniao {id_meeting}'
url = "https://easy-meeting.azurewebsites.net/external_url?meeting_day="+",".join(date_list)
email_server = create_email_server(user, psw)
for recipients_dict in list_of_recipients:
email_msg = create_email_body(user, recipients_dict, creator_name, subject, url)
run(email_server, email_msg)
dictionary = {"Status":"Sucesso ao enviar o email"}
except Exception as email_error:
dictionary = {"Status":"Erro ao enviar o email", "Error Message": email_error}
print(email_error)
dictionary = jsonify(dictionary)
return dictionary
full error code
2022-11-26T02:26:55.182403211Z [2022-11-26 02:26:55,162] ERROR in app: Exception on /send_email [POST]
2022-11-26T02:26:55.182438711Z Traceback (most recent call last):
2022-11-26T02:26:55.182445810Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
2022-11-26T02:26:55.182480710Z response = self.full_dispatch_request()
2022-11-26T02:26:55.182501610Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request
2022-11-26T02:26:55.182505610Z rv = self.handle_user_exception(e)
2022-11-26T02:26:55.182509810Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
2022-11-26T02:26:55.182513410Z rv = self.dispatch_request()
2022-11-26T02:26:55.182516910Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
2022-11-26T02:26:55.182520410Z return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2022-11-26T02:26:55.182524010Z File "/tmp/8dacf551bd5f657/api_run.py", line 55, in get_parameters
2022-11-26T02:26:55.182527510Z dictionary = jsonify(dictionary)
2022-11-26T02:26:55.182531010Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/json/__init__.py", line 342, in jsonify
2022-11-26T02:26:55.182534510Z return current_app.json.response(*args, **kwargs)
2022-11-26T02:26:55.182538010Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/json/provider.py", line 309, in response
2022-11-26T02:26:55.182541510Z f"{self.dumps(obj, **dump_args)}\n", mimetype=mimetype
2022-11-26T02:26:55.182545110Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/json/provider.py", line 230, in dumps
2022-11-26T02:26:55.182548610Z return json.dumps(obj, **kwargs)
2022-11-26T02:26:55.182551910Z File "/opt/python/3.10.4/lib/python3.10/json/__init__.py", line 238, in dumps
2022-11-26T02:26:55.182555410Z **kw).encode(obj)
2022-11-26T02:26:55.182558710Z File "/opt/python/3.10.4/lib/python3.10/json/encoder.py", line 199, in encode
2022-11-26T02:26:55.182562210Z chunks = self.iterencode(o, _one_shot=True)
2022-11-26T02:26:55.182565610Z File "/opt/python/3.10.4/lib/python3.10/json/encoder.py", line 257, in iterencode
2022-11-26T02:26:55.182569210Z return _iterencode(o, 0)
2022-11-26T02:26:55.182572510Z File "/tmp/8dacf551bd5f657/antenv/lib/python3.10/site-packages/flask/json/provider.py", line 122, in _default
2022-11-26T02:26:55.182576010Z raise TypeError(f"Object of type {type(o).__name__} is not JSON serializable")
2022-11-26T02:26:55.182579609Z TypeEr
I already tried to add the jsonify flask function, but it didn't work
The goal is to make the dictionary variable just like local
enter image description here
I am new to Locust (and Python) and am trying to load test an API. My Locust UI shows no stats whatsoever - no users hatched, no API calls, etc. I need help figuring out how to get them to show up on the UI so I can get the stats.
Here is my locust.py file:
from locust import HttpUser, SequentialTaskSet, task, constant
import finops_service.locust_files.finops_fx_load_testing as load_test
class FXTransaction(SequentialTaskSet):
def __init__(self, parent):
super().__init__(parent)
self.comp_data = dict()
self.rate_lock_response = dict()
self.transaction_response = dict()
self.fx_providerID = '57638f08-e938-48d7-accf-325b6728a9ee'
self.headers = {"Content-Type": "application/json"}
def on_start(self):
load_test.login(self),
#task
def get_company_data(self):
# Get company data
self.comp_data = load_test.get_company_data(self)
print("Company is: ", self.comp_data['company'])
print("Vendor is: ", self.comp_data['vendor'])
print("Payment Method is: ", self.comp_data['payment_method'])
print("Funding Method is: ", self.comp_data['funding_method'])
# #task
# def rate_lock(self):
print("Starting rate lock")
load_test.rate_lock(self)
print("This is the returned rate lock response:")
print(self.rate_lock_response)
# #task
# def approve_transaction(self):
# print("Starting transaction")
# load_test.approve_transaction(self)
# print("This is the returned transaction response:")
# print(self.transaction_response)
class MyUser(HttpUser):
wait_time = constant(1)
# weight = 1
host = "http://localhost:8080/PaymentService/base/"
tasks = [FXTransaction]
And here are my functions:
import json
import random
import uuid
from utils import json_util as json_util
enter code here
def login(self):
try:
with self.client.post(
"security/login",
headers={'Content-Type': 'application/json',
'Authorization': 'BASIC 0ee89b88-5c4b-4922-b1f9-c1584ab26e7e:12345'},
name=login,
timeout=55.6,
catch_response=True) as response:
if response.status_code != 200:
response.failure("Login failed")
else:
response.success()
print("Login succeeded")
self.headers.update({'if-Match': "{}".format(response.headers["ETag"])})
except ConnectionRefusedError:
print("The test could not connect to {}".format("http://localhost:8080/PaymentService/security/login"))
raise
def get_company_data(self):
comp_index = random.randrange(0, 9)
print("Random company data index is: ", comp_index)
try:
body = json.load(open("finops_service/locust_files/load_testing_data.json", 'r'))
comp_data = body['fx']['multipleCompanies_10']['company_data'][comp_index]
except ConnectionRefusedError:
print("The test could not connect to {}".format("http://localhost:8080/PaymentService/security/login"))
raise
return comp_data
def rate_lock(self):
body = json_util.get_json_object(
"/finops_service/locust_files/rate_lock_load_testing.json",
'fx'.lower(),
'valid_multiple_20'.lower()
)
body["debtorCompanyProfileId"] = self.comp_data['company']
i = 0
# there are 20 rate locks so need to update each request
while i < 20:
body["rateRequestPayments"][i]["creditorProfileId"] = self.comp_data['vendor']
body["rateRequestPayments"][i]["debtorProfileId"] = self.comp_data['company']
body["rateRequestPayments"][i]["paymentMethodId"] = self.comp_data['payment_method']
random_float_no = round(random.uniform(1.11, 999.99), 2)
body['rateRequestPayments'][i]['amount'] = random_float_no
i += 1
try:
print("RIGHT BEFORE RATE LOCK CALL")
with self.client.post("services/transaction/fx/rate/lock",
data=json.dumps(body),
headers=self.headers,
name="rate lock",
timeout=55.6,
catch_response=True) as response:
if "GENERIC_FAILURE" in response.text:
response.failure("FAIL")
print("Rate lock call failed")
else:
response.success()
print("rate lock succeeded")
print("Rate lock response is:.........", response)
print("Rate lock response TEXT is:.........", response.text)
self.rate_lock_response = response.text
except ConnectionRefusedError:
print("The test could not connect to {}".format(
"http://localhost:8080/PaymentService/services/transaction/fx/rate/lock"))
raise
When I run this, I can see that the self.client.post("services/transaction/fx/rate/lock"... is happening and either succeeding or failing. However, in the Locust UI, I show 0 users hatched (console shows I hatched 5) and 0 tasks shown.
I get this error each time I run which it looks related to my stats not showing up but I don't know why it is happening:
Traceback (most recent call last):
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/gevent/pywsgi.py", line 999, in handle_one_response
self.run_application()
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/gevent/pywsgi.py", line 945, in run_application
self.result = self.application(self.environ, self.start_response)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
response = self.handle_exception(e)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/locust/web.py", line 319, in wrapper
return view_func(*args, **kwargs)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/locust/util/cache.py", line 21, in wrapper
cache["result"] = func(*args, **kwargs)
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/site-packages/locust/web.py", line 208, in request_stats
"safe_name": escape(s.name, quote=False),
File "/Users/michellegautier/.pyenv/versions/3.7.4/lib/python3.7/html/__init__.py", line 19, in escape
s = s.replace("&", "&") # Must be done first!
AttributeError: 'function' object has no attribute 'replace'
2021-03-18T16:44:29Z {'REMOTE_ADDR': '::1', 'REMOTE_PORT': '49242', 'HTTP_HOST': 'localhost:8089', (hidden keys: 25)} failed with AttributeError
Can someone please help me figure out how to make my stats show up?
Thank you!!!!
The stats not showing up in Locust is because of that Exception you posted. If you resolve that and get to where your code runs without issues, you should see stats in the Locust UI.
There are a number of issues with your code.
I'm not sure if it's just an issue with pasting your code into SO, but your on_start() and get_company_data() functions (and the #task decorator) need to be indented so they're part of FXTransaction.
Also, trying to use self.client in login() or get_company_data() isn't going to work because they aren't part of a class so there's no such thing as self. You'd probably want to change self in that function to just client and then pass in self.client when you call it from the on_start() function.
But what's possibly your main issue is here:
with self.client.post(
"security/login",
headers={'Content-Type': 'application/json',
'Authorization': 'BASIC 0ee89b88-5c4b-4922-b1f9-c1584ab26e7e:12345'},
name=login,
…
name here is the name of the endpoint (URL) you're hitting. This is supposed to be a string but you're giving it your function login. So when the Locust code tries to do some thing with your name value that it wants to do, expecting it to be a string, it's going to fail because it's trying to do those on a function instead. That's what AttributeError: 'function' object has no attribute 'replace' means.
I think if you fix all of those things, Locust should work.
I am brand new to python as of about 3 days ago and I need to build a prototype python server for an iOS work project. I followed a tutorial for the initial build.
When I perform a "POST" request, I get an error. Here is my full code, followed by the error I'm receiving in the Terminal when a "POST" request is submitted.
PS. also fairly new to stack overflow so I'm sorry if I'm asking this question incorrectly
import flask
from flask import Flask, request, jsonify
from flask_uploads import UploadSet, IMAGES, configure_uploads
from flask import make_response
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
import turicreate as tc
import sys
from queue import Queue
import os
import uuid
import logging
from flask import send_from_directory
import threading
from marshmallow import fields
from marshmallow import post_load
app = Flask(__name__)
if __name__ == "__main__":
# Only for debugging while developing
app.run(host='0.0.0.0', debug=True)
#configure logging
logging.basicConfig( level=logging.DEBUG,
format='[%(levelname)s] - %(threadName)-10s : %(message)s')
#configure images destination folder
app.config['UPLOADED_IMAGES_DEST'] = './images'
images = UploadSet('images', IMAGES)
configure_uploads(app, images)
#configure sqlite database
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'facerecognition.sqlite')
db = SQLAlchemy(app)
ma = Marshmallow(app)
#model/users is a many to many relationship, that means there's a third #table containing user id and model id
users_models = db.Table('users_models',
db.Column("user_id", db.Integer, db.ForeignKey('user.id')),
db.Column("model_id", db.Integer, db.ForeignKey('model.version'))
)
# model table
class Model(db.Model):
version = db.Column(db.Integer, primary_key=True)
url = db.Column(db.String(100))
users = db.relationship('User', secondary=users_models)
# user table
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(300))
position = db.Column(db.String(300))
def __init__(self, name, position):
self.name = name
self.position = position
#user schema
class UserSchema(ma.Schema):
class Meta:
fields = ('id', 'name', 'position')
#model schema
class ModelSchema(ma.Schema):
version = fields.Int()
url = fields.Method("add_host_to_url")
users = ma.Nested(UserSchema, many=True)
# this is necessary because we need to append the current host to the model url
def add_host_to_url(self, obj):
return request.host_url + obj.url
#initialize everything
user_schema = UserSchema()
users_schema = UserSchema(many=True)
model_schema = ModelSchema()
models_schema = ModelSchema(many=True)
db.create_all()
#error handlers
#app.errorhandler(404)
def not_found(error):
return make_response(jsonify({'error': 'not found'}), 404)
#app.errorhandler(400)
def not_found(error):
return make_response(jsonify({'error': 'bad request'}), 400)
#register user
#app.route("/testImages/face-recognition/api/v1.0/user/register", methods=['POST'])
def register_user():
if not request.form or not 'name' in request.form:
return make_response(jsonify({'status': 'failed', 'error': 'bad request', 'message:' : 'Name is required'}), 400)
else:
name = request.form['name']
position = request.form.get('position')
if position is None:
position = ""
newuser = User(name, position)
db.session.add(newuser)
db.session.commit()
if 'photos' in request.files:
uploaded_images = request.files.getlist('photos')
save_images_to_folder(uploaded_images, newuser)
return jsonify({'status' : 'success', 'user' : user_schema.dump(newuser).data })
#function to save images to image directory
def save_images_to_folder(images_to_save, user):
for a_file in images_to_save:
# save images to images folder using user id as a subfolder name
images.save(a_file, str(user.id), str(uuid.uuid4()) + '.')
# get the last trained model
model = Model.query.order_by(Model.version.desc()).first()
if model is not None:
# increment the version
queue.put(model.version + 1)
else:
# create first version
queue.put(1)
#app.route("/testImages/face-recognition/api/v1.0/model/info" , methods=['GET'])
def get_model_info():
models_schema.context['request'] = request
model = Model.query.order_by(Model.version.desc()).first()
if model is None:
return make_response(jsonify({'status': 'failed', 'error': 'model is not ready'}), 400)
else:
return jsonify({'status' : 'success', 'model' : model_schema.dump(model).data})
#serve models
#app.route('/models/')
def download(filename):
return send_from_directory('models', filename, as_attachment=True)
def train_model():
while True:
#get the next version
version = queue.get()
logging.debug('loading images')
data = tc.image_analysis.load_images('images', with_path=True)
# From the path-name, create a label column
data['label'] = data['path'].apply(lambda path: path.split('/')[-2])
# use the model version to construct a filename
filename = 'Faces_v' + str(version)
mlmodel_filename = filename + '.mlmodel'
models_folder = 'models/'
# Save the data for future use
data.save(models_folder + filename + '.sframe')
result_data = tc.SFrame( models_folder + filename +'.sframe')
train_data = result_data.random_split(0.8)
#the next line starts the training process
model = tc.image_classifier.create(train_data, target='label', model='resnet-50', verbose=True)
db.session.commit()
logging.debug('saving model')
model.save( models_folder + filename + '.model')
logging.debug('saving coremlmodel')
model.export_coreml(models_folder + mlmodel_filename)
# save model data in database
modelData = Model()
modelData.url = models_folder + mlmodel_filename
classes = model.classes
for userId in classes:
user = User.query.get(userId)
if user is not None:
modelData.users.append(user)
db.session.add(modelData)
db.session.commit()
logging.debug('done creating model')
# mark this task as done
queue.task_done()
#configure queue for training models
queue = Queue(maxsize=0)
thread = threading.Thread(target=train_model, name='TrainingDaemon')
thread.setDaemon(False)
thread.start()```
Here is the Error I get In the Terminal
[DEBUG] - TrainingDaemon : loading images
127.0.0.1 - - [03/Jan/2020 11:52:39] "POST /testImages/face-recognition/api/v1.0/user/register HTTP/1.1" 500 -
[INFO] - Thread-3 : 127.0.0.1 - - [03/Jan/2020 11:52:39] "POST /testImages/face-recognition/api/v1.0/user/register HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/me/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/me/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/me/Desktop/GorillaFace/app.py", line 103, in register_user
save_images_to_folder(uploaded_images, newuser)
File "/Users/me/Desktop/GorillaFace/app.py", line 121, in save_images_to_folder
#app.route("/testImages/face-recognition/api/v1.0/model/info" , methods=['GET'])
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 1314, in decorator
self.add_url_rule(rule, endpoint, f, **options)
File "/Users/me/venv/lib/python3.7/site-packages/flask/app.py", line 90, in wrapper_func
"A setup function was called after the "
AssertionError: A setup function was called after the first request was handled. This usually indicates a bug in the application where a module was not imported and decorators or other functionality was called too late.
To fix this make sure to import all your view modules, database models and everything related at a central place before the application starts serving requests.
Exception in thread TrainingDaemon:
Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/Users/me/Desktop/GorillaFace/app.py", line 159, in train_model
model = tc.image_classifier.create(train_data, target='label', model='resnet-50', verbose=True)
File "/Users/me/venv/lib/python3.7/site-packages/turicreate/toolkits/image_classifier/image_classifier.py", line 225, in create
raise TypeError('"dataset" must be of type SFrame.')
TypeError: "dataset" must be of type SFrame.
What I've tried:
Like I said I'm still new to python so I don't know much about troubleshooting python code. The assertion error seems to be saying something was not imported correctly, so I re-installed, flask-uploads,Flask-SQLAlchemy,flask-marshmallow, marshmallow-sqlalchemy. I also made sure all the imports looked like it did in the tutorial.
Anyone have any ideas as to what I need to do to not get this error? what this code should do is allow me to post a name, position, and photos to the server which will then train a ML model to recognize the new photos.
Error shows in which line it has problem
model = tc.image_classifier.create(...)
and it has problem with
"dataset" must be of type SFrame.
so use at least print() and print(type(..)) to check variablse which you use in this line and see if you use SFrame.
Background Info
I'm trying to write my first flask / python REST API. So far, I have a GET that connects to a redis database and tries to convert a dictionary to json... and then return those results.
Problem
When I try to call jsonify on my dict object, it fails with the following error:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/localhost/htdocs/widgets/widgets.py", line 51, in get_profile
return jsonify(res_dict)
File "/usr/lib/python3.6/site-packages/flask/json.py", line 263, in jsonify
(dumps(data, indent=indent, separators=separators), '\n'),
File "/usr/lib/python3.6/site-packages/flask/json.py", line 123, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/lib/python3.6/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "/usr/lib/python3.6/json/encoder.py", line 430, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/usr/lib/python3.6/json/encoder.py", line 376, in _iterencode_dict
raise TypeError("key " + repr(key) + " is not a string")
TypeError: key b'email1' is not a string
The code looks like this:
20 def get_db_profile(mailbox):
21 """ connects to redis and queries for profile """
22 try:
23 my_redis = redis.Redis(connection_pool=POOL)
24 response = my_redis.hgetall(55555)
25 logging.info(response.keys())
26 logging.info(response.items())
27 #response = '123455555'
28 return response
29 except Exception as ex:
30 return "Error:", ex
47 #application.route("/widgets/api/<int:mailbox>", methods=['GET'])
48 def get_profile(mailbox):
49 res_dict = get_db_profile(mailbox)
50 # return application.response_class(jsonify(res_dict), content_type='application/json')
51 return jsonify(res_dict)
52 # return '12345'
53
I added some logging as you can see on line 25 to see what the keys() look like. this is what I see in the log file:
lab-1:/var/www/localhost/htdocs/widgets# cat /tmp/widgets.log
root - INFO - dict_keys([b'email1', b'email2'])
REDIS Data
This is how I created the redis data:
127.0.0.1:6379[5]> hmset 55555 email1 johndoe#hotmail.com email2 jd#yahoo.com
OK
127.0.0.1:6379[5]>
Questions
Should i be able to convert from dict object to json string?
What is the 'b' in the log files?
lab-1:/var/www/localhost/htdocs/widgets# cat /tmp/widgets.log
root - INFO - dict_keys([b'email1', b'email2'])
root - INFO - dict_items([(b'email1', b'johndoe#hotmail.com'), (b'email2', b'jd#yahoo.com')])
How do I send back a proper JSON response?
EDIT 1
I found this question/answer: How to parse python b' string containing dict
and based on that I tried to change my logic to look like this:
47 #application.route("/pvmailprofiles/api/<int:mailbox>", methods=['GET'])
48 def get_profile(mailbox):
49 res_dict = get_db_profile(mailbox)
50 # return application.response_class(jsonify(res_dict), content_type='application/json')
51 #return jsonify(res_dict)
52 return res_dict[b'email1']
53 # return '12345'
54
Line 52 - you can see that as a test I've hardcoded the 'b' and then the key name. And it works - it returns the specific email address.
But I need to find a way to "just" convert everything to json and return to callee.
Have you tried first importing json like so
import json
and then doing
json.dumps(myDict)
and then to load it again if needed.
json.loads(response.text)
I am writing unit tests for a Flask app using unittest and using mockito to mock the services which require the server to be up and running.
However I am unable to figure out how to mock the server. I am getting some error.
Here are the code snippets:
node.py
#app.route('/transaction', methods = ["POST"])
def transaction():
req = request.get_json()
print("\nreq")
i = bitcoin.addTransaction(req)
if req['purpose'] != 'redeem':
users = getColumn(sheet_u,0)
index = users.index(req['recipient'])
user = getRow(sheet_u,index)
hasPoints = int(user[labels_u.index("POINTS")])
hasPoints += req['amount']
setCell(sheet_u, index, labels_u.index("POINTS"), hasPoints)
wb.save("db.xlsx")
else:
users = getColumn(sheet_u,0)
index = users.index(req['sender'])
user = getRow(sheet_u,index)
hasPoints = int(user[labels_u.index("POINTS")])
hasPoints -= req['amount']
setCell(sheet_u, index, labels_u.index("POINTS"), hasPoints)
wb.save("db.xlsx")
r = requests.get(url = bitcoin.currentNodeUrl+"/save")
return jsonify({"note":"New transaction will be added to block "+str(i)})
#app.route('/transaction/broadcast', methods = ["POST"])
def transactionBroadcast():
check = True
failure = ""
req = request.get_json()
# Hiding some methods that change the value of check flag
if check:
newTransaction = bitcoin.createNewTransaction(req['amount'],req['sender'],req['recipient'],req['purpose'],time.time())
bitcoin.addTransaction(newTransaction)
promises = []
for node in bitcoin.networkNodes:
response = ""
try:
r = requests.post(url=node+"/transaction",json=newTransaction) #this line throws error
response = r.text #program control doesn't come here
print(newTransaction)
print("\n")
print(response)
except requests.exceptions.ConnectionError:
response = "{\"note\":\"The node is unavailable. It will sync itself later on.\"}"
if 'note' in json.loads(response):
promises.append(json.loads(response))
if len(promises) == len(bitcoin.networkNodes):
r = requests.get(url = bitcoin.currentNodeUrl+"/save")
return jsonify({"note":"Transaction broadcasted successfully"})
else:
return jsonify({"fail":"Invalid transaction. "+failure})
if __name__ == '__main__':
bitcoin = Blockchain(sys.argv[1])
app.run(port=sys.argv[1])
test_t2.py
from node import app
from flask import json
import unittest
import node
import requests
from mockito import when, mock, unstub
class TestSearch(unittest.TestCase):
print("inside TEST class")
def test_transactionBroadcast(self):
node.bitcoin = node.Blockchain('5000')
response = mock({"note": "New transaction will be added to block 5"})
when(requests).post('http://localhost:5001/transaction', strict = False ).thenReturn(response)
data1 = {
"sender" : "ph1000005",
"recipient" : "TIAA",
"amount" : 1,
"purpose" : "redeem"
}
response = app.test_client().post(
'/transaction/broadcast',
json = data1,
content_type='application/json',
)
print("running TransactionBroadcast test")
print(response)
self.assertEqual(response.get_json(), {
# 'fail': "Invalid transaction. This user doesn't exist. Check userid again."
"note": "Transaction broadcasted successfully"
})
unstub()
if __name__ == '__main__':
unittest.main()
error log
C:\POC>coverage run test_t2.py
inside TEST class
[2018-12-12 19:58:22,181] ERROR in app: Exception on /transaction/broadcast [POST]
Traceback (most recent call last):
File "c:\program files\python36\lib\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "c:\program files\python36\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "c:\program files\python36\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "c:\program files\python36\lib\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "c:\program files\python36\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "c:\program files\python36\lib\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\POC\node.py", line 230, in transactionBroadcast
r = requests.post(url=node+"/transaction",json=newTransaction)
File "c:\program files\python36\lib\site-packages\mockito\mocking.py", line 89, in new_mocked_method
self, method_name, *args, **kwargs)
File "c:\program files\python36\lib\site-packages\mockito\mocking.py", line 44, in remembered_invocation_builder
return invoc(*args, **kwargs)
File "c:\program files\python36\lib\site-packages\mockito\invocation.py", line 103, in __call__
""" % (self, "\n ".join(str(invoc) for invoc in stubbed_invocations)))
mockito.invocation.InvocationError:
You called
post(url='http://localhost:5001/transaction', json={'amount': 1, 'purpose': 'redeem', 'recipient': 'TIAA', 'sender': 'ph1000005', 'timestamp': 1544624902.1440911, 'transactionId': 'd919ffb4080f47e890957f3f3edc97a1'}),
which is not expected. Stubbed invocations are:
post('http://localhost:5001/transaction', strict=False)
(Set strict to False to bypass this check.)
running TransactionBroadcast test
<Response streamed [500 INTERNAL SERVER ERROR]>
F
======================================================================
FAIL: test_transactionBroadcast (__main__.TestSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_t2.py", line 49, in test_transactionBroadcast
"note": "Transaction broadcasted successfully"
AssertionError: None != {'note': 'Transaction broadcasted successfully'}
----------------------------------------------------------------------
Ran 1 test in 0.682s
FAILED (failures=1)
Now, I'm not sure what am I doing wrong. there is an internal route call to /transaction in the route /transaction/broadcast which is failing.
Since this is a unit testing, it doesn't require the server to be up and running. and the route is calling localhost
Any ideas on how to deal with it?
Basically, the stub error message tells you that your stubbing signature
when(requests).post('http://localhost:5001/transaction', strict = False ).thenReturn(response)
is wrong.
Note 1: when using strict it must go into when like so: when(requests, strict=False). But not being strict is usually not what you want.
Instead you fix your stubbing, e.g.:
when(requests).post(url='http://localhost:5001/transaction', ...).thenReturn(response)
# or for python2
from mockito import kwargs
when(requests).post(url='http://localhost:5001/transaction', **kwargs).thenReturn(response)