I am launching REST API build in python.
I am inputting a list as input and getting the required data.
example:
your.api.com/birth?name=James&date=2015-02-01&name=Robert&date=2020-01-01
from flask import request
#app.route('/birth')
def birth():
names = request.form.getlist('name')
dates = request.form.getlist('date')
As the number of inputs I have is huge, the end point URL is becoming huge. Is there any way to do the same using PUT or POST where I dump a doc in some format (say json) as my input?
Sure, put the request fields in a JSON and go through the dictionary
#app.route('/birth', methods=['POST'])
def birth():
data = request.get_json()
names = data["names"]
# etc.
When the input grows it is suggested to use a POST call rather than GET call.
from flask import request
#app.route('/birth')
def birth():
names = request.form.getlist('name')
dates = request.form.getlist('date')
Convert /birth to the following
from flask import request
#app.route('/birth', methods=['POST'])
def birth():
input = request.get_json()
# <Do the processing>
And in the POST call from client-side use a JSON like the following
{
name: [<array of values>],
date: [<array of values>]
}
Related
This is my first time using Flask to build an API.
I have a CSV file that I'm reading and storing into a dataframe and then being converted to a dictionary in order to become a response for a Flask API using flask_restful to show a JSON output.
The CSV file has around 230000 rows, so when I try to connect to the endpoint it takes to much time to load the data. The data is only fully loaded if I use parameters to filter by date, so the dataframe can show less rows.
If I try to use Postman to test the endpoint I can see that the full load is 120mb of data (only if I change the settings to show more than 50mb), so Power BI or any other tool is not showing all the data as well.
This is the response I get with full data from the endpoint without parameters (it takes to much time to load):
So my question is, is there any other way to show the JSON output from Flask and connect in other tools with this amount of rows?
Endpoint with full data: https://masterdimensions.herokuapp.com/dimensioncalendar
Endpoint with parameters: https://masterdimensions.herokuapp.com/dimensioncalendar?start_dt=2020-01-01&end_dt=2021-12-31
My application:
# app.py
from flask import Flask
from flask_restful import Resource, Api, reqparse
import pandas as pd
import os
import pathlib
# Launching an app flask
app = Flask(__name__)
api = Api(app, default_mediatype='application/json')
# API class for Index page
class Index(Resource):
def get(self):
index_data = [
{
"href": "/dimensioncalendar",
"name": "Dimension Calendar",
"description": "Dimension Calendar endpoint with date attributes"
}
]
return (index_data), 200
# API class for DimensionCalendar page
class DimensionCalendar(Resource):
def get(self):
# Initialize reparse from Flask
parser = reqparse.RequestParser()
# Add arguments
parser.add_argument('start_dt')
parser.add_argument('end_dt')
parser.add_argument('lang')
args = parser.parse_args() # parse arguments to dictionary
# Read CSV file
data = pd.read_csv(os.path.join(os.path.sep, pathlib.Path(__file__).parent.resolve(), "static", "csv", "DimensionCalendar.csv"))
# If parameters on API is not null then filter dataframe by start date and end date
if args['start_dt'] is not None and args['end_dt'] is not None:
data = data.loc[(data['DT_SHORTDATE'] >= str(args['start_dt'])) & (data['DT_SHORTDATE'] <= str(args['end_dt']))]
elif args['start_dt'] is not None and args['end_dt'] is None:
data = data.loc[data['DT_SHORTDATE'] == str(args['start_dt'])]
if args['lang'] == 'pt-BR':
data.columns = ['DT_DATA', 'NR_ANO', 'NR_MES', 'NR_DIA', 'NM_MES', 'NM_DIADASEMANA', 'NR_MESANO', 'NM_MESANO', 'NM_DATALONGA', 'NR_DIADASEMANA', 'NR_DIADOANO', 'NR_SEMESTRE', 'NM_SEMESTRE', 'NR_QUADRIMESTRE', 'NR_TRIMESTRE', 'NM_TRIMESTRE', 'NR_BIMESTRE', 'NM_BIMESTRE', 'NR_ANORELATIVO', 'NR_MESRELATIVO', 'NR_DIARELATIVO', 'NR_DATA', 'NR_ANOMES', 'NR_ANOTRIMESTRE', 'IC_DIAUTIL', 'IC_FINALDESEMANA', 'IC_FERIADO', 'IC_ANOATUAL', 'IC_MESATUAL', 'IC_DATAATUAL', 'IC_ANOPASSADO', 'IC_MESPASSADO', 'IC_ONTEM', 'LINDATA', 'LINORIGEM']
# Convert dataframe to dictionary
data = data.to_dict()
# Return data and 200 OK code
return (data), 200
# Case the page is not found
#app.errorhandler(404)
def page_not_found(e):
return "<h1>404</h1><p>The resource could not be found.</p>", 404
# '/dimensioncalendar' is our entry point for DimensionCalendar
api.add_resource(DimensionCalendar, '/dimensioncalendar')
# '/Index' is our entry point for Index
api.add_resource(Index, '/')
# Run our Flask app
if __name__ == '__main__':
app.run()
I have a function that generates all video URLs of some user on Twitch
def get_videos(cursor=None): # functiont to retrieve all vod URLs possible, kinda slow for now
params_get_videos = {('user_id', userid_var)} # params for request.get
if cursor is not None: # check if there was a cursor value passed
params_get_videos = list(params_get_videos) + list({('after', cursor)}) # add another param for pagination
url_get_videos = 'https://api.twitch.tv/helix/videos' # URL to request data
response_get_videos = session.get(url_get_videos, params=params_get_videos, headers=headers) # get the data
reponse_get_videos_json = response_get_videos.json() # parse and interpret data
file = open(MyUsername+" videos.txt", "w")
for i in range(0, len(reponse_get_videos_json['data'])): # parse and interpret data
file.write(reponse_get_videos_json['data'][i]['url'] +'\n') # parse and interpret data
if 'cursor' in reponse_get_videos_json['pagination']: # check if there are more pages
get_videos(reponse_get_videos_json['pagination']['cursor']) # iterate the function until there are no more pages
this works perfectly fine on its own (with other functions) but whenever i try to call it from a dummy flask server like this
from flask import Flask
from flask import render_template
from flask import request
from main import *
app = Flask(__name__)
#app.route('/')
def hello_world():
return render_template("hello.html")
#app.route('/magic', methods=['POST', 'GET'])
def get_username():
username = request.form.get('username')
get_videos()
return ("Success")
It no longer acts recursive and only prints first 20 values. What am I doing wrong?
I'm new also so I don't have enough reputation to make a comment, I have to post an answer. I'm confused by your get_username method, once you grab the username from the form you're not sending it anywhere? It looks like on your get_videos method you are maybe hard coding the username by saving a variable called MyUsername outside of this method?
What you should do imo is send the username you grab from the form, do your get_videos method like this.
get_videos(username)
And change your other method to this
def get_videos(MyUsername, cursor=None):
I'm getting a POST request inside a Flask app from Slack. The request is sent when a user presses on an interactive message button. According to Slack docs I must extract the body of the request to verify the signature.
My computed signature doesn't match the one sent by Slack, though.
In fact, the body of the request comes as some encoded string. The string is actually an encoded dictionary instead of a query str parameters, as expected.
Here's the beginning of my view:
#app.route('/register', methods=['POST'])
def register_visit():
data = request.get_data()
signature = request.headers.get('X-Slack-Signature', None)
timestamp = request.headers.get('X-Slack-Request-Timestamp', None)
signing_secret = b'aaaaaaaaaaaaaaaa'
# old message, ignore
if round(actual_time.time() - float(timestamp)) > 60 * 5:
return
concatenated = ("v0:%s:%s" % (timestamp, data)).encode('utf-8')
computed_signature = 'v0=' + hmac.new(signing_secret, msg=concatenated, digestmod=hashlib.sha256).hexdigest()
if hmac.compare_digest(computed_signature, signature):
...
I've tried to format the received data to make it look like:
token=fdjkgjl&user_id=1234... but I am not aware of all of the necessary parameters that have to be present in the data.
Any ideas are highly appreciated.
The body of the message is following - after being URL decoded (note I've modified possibly sensitive data):
b'payload={"type":"interactive_message","actions":
[{"name":"yes_button","type":"button","value":"236"}],"callback_id":"visit_button","team":{"id":"fffff","domain":"ffff"},"channel":{"id":"ffff","name":"directmessage"},"user":{"id":"ffffff","name":"fffft"},"action_ts":"1540403943.419120","message_ts":"1541403949.000100","attachment_id":"1","token":"8LpjBuv13J7xAjhl2lEajoBU","is_app_unfurl":false,"original_message":{"text":"Test","bot_id":"DDDDDDDDD","attachments":[{"callback_id":"visit_button","text":"Register","id":1,"color":"3AA3E3","actions":[{"id":"1","name":"yes_button","text":"Yes","type":"button","value":"236","style":""}],"fallback":"Register"}],"type":"message","subtype":"bot_message","ts":"1540413949.000100"},"response_url":"https://hooks.slack.com/actions/ffffff/ffffff/tXJjx1XInaUhrikj6oEzK08e","trigger_id":"464662548327.425084163429.dda35a299eedb940ab98dbb9386b56f0"}'
The reason you are getting the "garbled" data is that you are using request.get_data(). That method will return the raw data of a request, but not do any decoding for you.
Much more convenient is to use request.form.get('payload'), which will directly give you the JSON string of the request object. You can then convert that into a dict object with json.loads() to process it further in your app.
Note that the format you received is the correct format for interactive messages. You will not get a query string (e.g. "token=abc;user_id?def...") as you suggested (like for slash command requests). Interactive message request will always contain the request as JSON string in a payload form property. See here for reference.
Here is a simple working example, which will reply a greeting to the user that pressed the button. It will work directly with Slack, but I recommend using Postman to test it.
#app.py
from flask import Flask, request #import main Flask class and request object
import json
app = Flask(__name__) #create the Flask app
#app.route('/register', methods=['POST'])
def register_visit():
slack_req = json.loads(request.form.get('payload'))
response = '{"text": "Hi, <#' + slack_req["user"]["id"] + '>"}'
return response, 200, {'content-type': 'application/json'}
if __name__ == '__main__':
app.run(debug=True, port=5000) #run app in debug mode on port 5000
OK, the issue wasn't related to how Slack sends me the message. It was about misunderstanding which data comes as bytes and which data is unicode. The culprit was string formatting in my case - the line concatenated = ("v0:%s:%s" % (timestamp, data)).encode('utf-8') should have been concatenated = (b"v0:%b:%b" % (timestamp.encode("utf-8"), data)). Data is already bytes, timestamp meanwhile is unicode.
Cannot believe I've banged my head on this for hours -_-
#app.route('/register', methods=['POST'])
def register_visit():
data = request.get_data()
signature = request.headers.get('X-Slack-Signature', None)
timestamp = request.headers.get('X-Slack-Request-Timestamp', None)
signing_secret = b'aaaaaaaaaaaaaaaa'
# old message, ignore
if round(actual_time.time() - float(timestamp)) > 60 * 5:
return
concatenated = (b"v0:%b:%b" % (timestamp.encode("utf-8"), data))
computed_signature = 'v0=' + hmac.new(signing_secret, msg=concatenated,
digestmod=hashlib.sha256).hexdigest()
if hmac.compare_digest(computed_signature, signature):
...
This worked for me
from urllib import parse
parsed_text = parse.unquote('your bytes text here')
I'm writing a very simple demo webapp, and I can't seem to pass a json object from js to python using ajax.
I've tried a number of suggestions from people on so with similar problems, such as using .get_json() instead of .json, passing the object without using JSON.stringify in the javascript, etc.
Any idea what piece I'm missing here?
Javascript
var run_method = function(){
var data1 = {"word":"hello"}
console.log("Before " + data1);
$.ajax({
url : "/examplemethod",
type : "POST",
data : data1//JSON.stringify(data1)
})
.done(function(data){
var data = JSON.parse(data);
console.log(data);
});
}
Python
#app.route("/examplemethod", methods=['POST', 'GET'])
def example_method():
global data
if request.method == 'POST':
print request
data = request.json
print "data", data
return "after "+ data["word"]
Every variation I have tried of this gives a 500 Error, and
TypeError: 'NoneType' object has no attribute 'getitem'
Obviously, that is because data is supposed to be an dictionary/json, not None. But how to I get it to return as a dictionary?
Because you are not sending JSON, to the flask app (see this and this). Your current code results in a standard urlencoded form post. Which in turn results in an entry being populated in request.form
request.form.get('word')
Switch to a json post as per the guidelines in the above Q&As to access the data through request.json.
the data is likely not flowing to json if you are getting None, so you should jsonify the data. It will be coming in in the form of form.
from flask import jsonify
#app.route("/examplemethod", methods=['POST'])
def example_method():
data = jsonify(request.form).json
print(data) #will be in the form a json dict
return data['foo-key'] #can retrieve specific items with their key-pairs
In order to test a Flask application, I got a flask test client POSTing request with files as attachment
def make_tst_client_service_call1(service_path, method, **kwargs):
_content_type = kwargs.get('content-type','multipart/form-data')
with app.test_client() as client:
return client.open(service_path, method=method,
content_type=_content_type, buffered=True,
follow_redirects=True,**kwargs)
def _publish_a_model(model_name, pom_env):
service_url = u'/publish/'
scc.data['modelname'] = model_name
scc.data['username'] = "BDD Script"
scc.data['instance'] = "BDD Stub Simulation"
scc.data['timestamp'] = datetime.now().strftime('%d-%m-%YT%H:%M')
scc.data['file'] = (open(file_path, 'rb'),file_name)
scc.response = make_tst_client_service_call1(service_url, method, data=scc.data)
Flask Server end point code which handles the above POST request is something like this
#app.route("/publish/", methods=['GET', 'POST'])
def publish():
if request.method == 'POST':
LOG.debug("Publish POST Service is called...")
upload_files = request.files.getlist("file[]")
print "Files :\n",request.files
print "Upload Files:\n",upload_files
return render_response_template()
I get this Output
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>)])
Upload Files:
[]
If I change
scc.data['file'] = (open(file_path, 'rb'),file_name)
into (thinking that it would handle multiple files)
scc.data['file'] = [(open(file_path, 'rb'),file_name),(open(file_path, 'rb'),file_name1)]
I still get similar Output:
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>), ('file', <FileStorage: u'Second_XML.xml' ('application/xml')>)])
Upload Files:
[]
Question:
Why request.files.getlist("file[]") is returning an empty list?
How can I post multiple files using flask test client, so that it can be retrieved using request.files.getlist("file[]") at flask server side ?
Note:
I would like to have flask client I dont want curl or any other client based solutions.
I dont want to post single file in multiple requests
Thanks
Referred these links already:
Flask and Werkzeug: Testing a post request with custom headers
Python - What type is flask.request.files.stream supposed to be?
You send the files as the parameter named file, so you can't look them up with the name file[]. If you want to get all the files named file as a list, you should use this:
upload_files = request.files.getlist("file")
On the other hand, if you really want to read them from file[], then you need to send them like that:
scc.data['file[]'] = # ...
(The file[] syntax is from PHP and it's used only on the client side. When you send the parameters named like that to the server, you still access them using $_FILES['file'].)
Lukas already addressed this,just providing these info as it may help someone
Werkzeug client is doing some clever stuff by storing requests data in MultiDict
#native_itermethods(['keys', 'values', 'items', 'lists', 'listvalues'])
class MultiDict(TypeConversionDict):
"""A :class:`MultiDict` is a dictionary subclass customized to deal with
multiple values for the same key which is for example used by the parsing
functions in the wrappers. This is necessary because some HTML form
elements pass multiple values for the same key.
:class:`MultiDict` implements all standard dictionary methods.
Internally, it saves all values for a key as a list, but the standard dict
access methods will only return the first value for a key. If you want to
gain access to the other values, too, you have to use the `list` methods as
explained below.
getList call looks for a given key in the "requests" dictionary. If the key doesn't exist, it returns empty list.
def getlist(self, key, type=None):
"""Return the list of items for a given key. If that key is not in the
`MultiDict`, the return value will be an empty list. Just as `get`
`getlist` accepts a `type` parameter. All items will be converted
with the callable defined there.
:param key: The key to be looked up.
:param type: A callable that is used to cast the value in the
:class:`MultiDict`. If a :exc:`ValueError` is raised
by this callable the value will be removed from the list.
:return: a :class:`list` of all the values for the key.
"""
try:
rv = dict.__getitem__(self, key)
except KeyError:
return []
if type is None:
return list(rv)
result = []
for item in rv:
try:
result.append(type(item))
except ValueError:
pass
return result