I am about to implement a Push Task queue in Flask/Google App Engine.
Essentially I would like to POST to the API and execute the underlying work in the task queue.
The initial entry point is a REST API (flask_restful)
class FTRecordsAPI(Resource):
def post(self):
arguments = self.reqparser.parse_args()
json_records = arguments.get('records')
user = User.query(...).get()
if user:
taskqueue.add(url='/worker/', params={'user': user})
return '', 201
else:
return '', 401
The worker is defined as a view in the url.py:
app.add_url_rule('/worker', 'worker',
view_func=csrf_protect.exempt(TaskView.as_view('taskView')))
And the TaskView is:
from flask.globals import request
class TaskView(MethodView):
def post(self):
user = request.json['user']
return "OK"
Strangely when I debug in the TaskView nowhere in the request object is any trace of the user object I sent to the /worker. However I find in there the records object which was from the previous call ?!
What am I missing please?
Try:
taskqueue.add(url='/worker', params={'user': user}, method="POST")
and
user = request.form.get('user')
As marcadian pointed out, taskqueue uses POST by default, so perhaps you need the request.form to access the POST vars.
Sorry for the thread resurrection, but should you want to use json directly you can do the following for instance:
taskstop = taskqueue.add(url='/stop_std_watcher',
target='worker',
payload=json.dumps({'watcherjson': client.client_watcher}),
method='POST',
headers={'Content-Type': 'application/json'})
the important bits being to add the headers and payload kwargs
On the receiving end, you got then:
#app.route('/stop_std_watcher', methods=['POST'])
def stop_std_watcher():
# curl -i -H "Content-Type: application/json" -X POST -d #stop.json http://localhost:8081/stop_std_watcher
watcherjson = request.json['watcherjson']
I added the curl in comments should you want to test your post route manually, can be useful
Related
I am writing these two following function in my code to be able to process incoming messages and respond back to the user on Messenger via a bot:
#app.route('/', methods=['post'])
def webhook():
# endpoint for processing incoming messaging events
data = request.get_json()
print(data) # you may not want to log every incoming message in production, but it's good for testing
if data["object"] == "page":
for entry in data["entry"]:
for messaging_event in entry["messaging"]:
if messaging_event.get("message"): # someone sent us a message
sender_id = messaging_event["sender"]["id"] # the Facebook ID of the person sending you the message
recipient_id = messaging_event["recipient"]["id"] # the recipient's ID, which should be your page's facebook ID
message_text = messaging_event["message"]["text"] # the message's text
responseai = response(message_text, sender_id)
send_message(sender_id, responseai)
if messaging_event.get("delivery"): # delivery confirmation
pass
if messaging_event.get("optin"): # optin confirmation
pass
if messaging_event.get("postback"): # user clicked/tapped "postback" button in earlier message
pass
return "Ok", 200
#app.route('/', methods=['GET'])
def verify():
# when the endpoint is registered as a web hook, it must echo back
# the 'hub.challenge' value it receives in the query arguments
if request.args.get("hub.mode") == "subscribe" and request.args.get("hub.challenge"):
if not request.args.get("hub.verify_token") == os.environs["VERIFY_TOKEN"]:
return "Verification token mismatch", 403
return request.args["hub.challenge"], 200
return "Hello World", 200
When I access my localhost:5000 where my Flask is, only Hello World appears on the browser. How would I know that the function web-hook is working? Should it also display the 'Ok'?
Your browser will not send a POST request, not without some additional HTML and Javascript coding. The simplest way to test if your hook works is by using the curl command-line client.
It can send both GET and POST requests. Testing if your GET handler works correctly:
curl -X GET "localhost:5000/?hub.verify_token=<YOUR_VERIFY_TOKEN>&hub.challenge=CHALLENGE_ACCEPTED&hub.mode=subscribe"
should produce CHALLENGE_ACCEPTED as the output. Then test the POST handler with:
curl -H "Content-Type: application/json" -X POST "localhost:5000/" -d '{"sender":{"id":"<PSID>"}, "recipient":{"id":"<PAGE_ID>"}, "timestamp":1458692752478, "message":{"mid":"mid.1457764197618:41d102a3e1ae206a38", "text":"hello, world!", "quick_reply": {"payload": "<DEVELOPER_DEFINED_PAYLOAD>"}}}'
See the Setting Up Your Webhook section of the Messenger Platform Getting Started documentation, and the message received event for details on what to expect when handling a message event.
Another option is to write Python tests to cover the same:
import os
import pytest
import your_flask_module
#pytest.fixture
def client():
your_flask_module.app.config['TESTING'] = True
yield your_flask_module.app.test_client()
def test_register(client):
args = {
'hub.verify_token': os.environ["VERIFY_TOKEN"],
'hub.challenge': 'CHALLENGE_ACCEPTED',
'hub.mode': 'subscribe',
}
rv = client.get('/', params=args)
assert b'CHALLANGE_ACCEPTED' in rv.data
def test_message_event(client):
event = {
"sender": {"id": "<PSID>"},
"recipient": {"id":"<PAGE_ID>"},
"timestamp": 1458692752478,
"message": {
"mid": "mid.1457764197618:41d102a3e1ae206a38",
"text": "hello, world!",
"quick_reply": {
"payload": "<DEVELOPER_DEFINED_PAYLOAD>"
}
}
}
rv = client.post('/', json=event)
assert rv.status_code == 200
My API works fine and I see a 200 status when I test it using Postman. However I'm trying to access it using a Kivy application but I'm seeing a 400 response from the server after some waiting or quitting the app. By the way when testing with Postman I specify header as Content-Type: application/json and in body I see my parameters
{
"search_text": "Hello",
"num_results": 1
}
being sent as raw data.
My code
def search(self, search_text):
header = {'Content-Type':'application/json'}
req = UrlRequest('http://127.0.0.1:5000/search',req_body={"search_text": search_text,"num_results": 1},on_success=Test.got_json,req_headers=header)
print("Search method called")
#staticmethod
def got_json(req,result):
print(result)
Kivy docs say that you don't have to specify a method as this would send a POST request so I've not specified that here
Edit: The code for the server is kind of irrelevant for my issue here so I've removed it
UrlRequest should be passed a str object as request body. You can serialize the request dictionary as a string object by dumping it. Pass this dumped dictionary as the request body to UrlRequest.
import json
req_body=json.dumps({'search_text': search_text, 'num_results': 1})
req = UrlRequest(
'http://127.0.0.1:5000/search',
req_body=req_body,
on_success=Test.got_json,
req_headers=header)
req_body is a string parameter, might be a bit confusing as req_headers is a dict. You can use:
req_body=json.dumps({"search_text": search_text,"num_results": 1})
I am new to backend development using Flask and I am getting stuck on a confusing problem. I am trying to send data to an endpoint whose Timeout session is 3000 ms. My code for the server is as follows.
from flask import Flask, request
from gitStat import getGitStat
import requests
app = Flask(__name__)
#app.route('/', methods=['POST', 'GET'])
def handle_data():
params = request.args["text"].split(" ")
user_repo_path = "https://api.github.com/users/{}/repos".format(params[0])
first_response = requests.get(
user_repo_path, auth=(
'Your Github Username', 'Your Github Password'))
repo_commits_path = "https://api.github.com/repos/{}/{}/commits".format(params[
0], params[1])
second_response = requests.get(
repo_commits_path, auth=(
'Your Github Username', 'Your Github Password'))
if(first_response.status_code == 200 and params[2] < params[3] and second_response.status_code == 200):
values = getGitStat(params[0], params[1], params[2], params[3])
response_url = request.args["response_url"]
payload = {
"response_type": "in_channel",
"text": "Github Repo Commits Status",
"attachments": [
{
"text": values
}
]
}
headers = {'Content-Type': 'application/json',
'User-Agent': 'Mozilla /5.0 (Compatible MSIE 9.0;Windows NT 6.1;WOW64; Trident/5.0)'}
response = requests.post(response_url, json = test, headers = headers)
else:
return "Please enter correct details. Check if the username or reponame exists, and/or Starting date < End date. \
Also, date format should be MM-DD"
My server code takes arguments from the request it receives and from that request's JSON object, it extracts parameters for the code. This code executes getGitStats function and sends the JSON payload as defined in the server code to the endpoint it received the request from.
My problem is that I need to send a text confirmation to the endpoint that I have received the request and data will be coming soon. The problem here is that the function, getGitStats take more than a minute to fetch and parse data from Github API.
I searched the internet and found that I need to make this call asynchronous and I can do that using queues. I tried to understand the application using RQ and RabbitMQ but I neither understood nor I was able to convert my code to an asynchronous format. Can somebody give me pointers or any idea on how can I achieve this?
Thank you.
------------Update------------
Threading was able to solve this problem. Create another thread and call the function in that thread.
If you are trying to have a async task in request, you have to decide whether you want the result/progress or not.
You don't care about the result of the task or if there where any errors while processing the task. You can just process this in a Thread and forget about the result.
If you just want to know about success/fail for the task. You can store the state of the task in Database and query it when needed.
If you want progress of the tasks like (20% done ... 40% done). You have to use something more sophisticated like celery, rabbitMQ.
For you i think option #2 fits better. You can create a simple table GitTasks.
GitTasks
------------------------
Id(PK) | Status | Result
------------------------
1 |Processing| -
2 | Done | <your result>
3 | Error | <error details>
You have to create a simple Threaded object in python to processing.
import threading
class AsyncGitTask(threading.Thread):
def __init__(self, task_id, params):
self.task_id = task_id
self.params = params
def run():
## Do processing
## store the result in table for id = self.task_id
You have to create another endpoint to query the status of you task.
#app.route('/TaskStatus/<int:task_id>')
def task_status(task_id):
## query GitTask table and accordingly return data
Now that we have build all the components we have to put them together in your main request.
from Flask import url_for
#app.route('/', methods=['POST', 'GET'])
def handle_data():
.....
## create a new row in GitTasks table, and use its PK(id) as task_id
task_id = create_new_task_row()
async_task = AsyncGitTask(task_id=task_id, params=params)
async_task.start()
task_status_url = url_for('task_status', task_id=task_id)
## This is request you can return text saying
## that "Your task is being processed. To see the progress
## go to <task_status_url>"
On GAE, I need to make some REST calls to a PiCloud server, then create an output page based on PiCloud return values. However, it will take several minutes for PiCloud to process the model. Thus I am wondering if I could create a 'loading' page first, and after finishing calculation, present the real output page.
In detail, the questions is how do I keep checking the status of my REST service and then generate different HTML pages based upon.
I appreciate any suggestions and comments!
PS: jQuery BlockUI seems to be good example, but it requires to estimate the timeout duration, which I could not guess...
Function to Call REST Service:
def get_jid(pdf_t, pdf_nop, pdf_p):
response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers)
jid= json.loads(response.content)['jid']
output_st = "running"
while output_st!="done":
response_st = urlfetch.fetch(url='https://api.picloud.com/job/?jids=%s&field=status' %jid, headers=http_headers)
output_st = json.loads(response_st.content)['info']['%s' %jid]['status']
url_val = 'https://api.picloud.com/job/result/?jid='+str(jid)
response_val = urlfetch.fetch(url=url_val, method=urlfetch.GET, headers=http_headers)
output_val = json.loads(response_val.content)['result']
return(jid, output_st, output_val)
Generate HTML Page:
class pdfPage_loading(webapp.RequestHandler):
def post(self):
final_res=get_jid(pdf_t, pdf_nop, pdf_p)[2]
html = html + template.render(templatepath + 'popup_pdf_eco.html', {
'title':'Ubertool',
'model_page':'',
'model_attributes':'Please wait','text_paragraph':''})
self.response.out.write(html)
class pdfPage_done(webapp.RequestHandler):
def post(self):
final_res=get_jid(pdf_t, pdf_nop, pdf_p)[2]
html = html + template.render(templatepath + 'popup_pdf_eco.html', {
'title':'Ubertool',
'model_page':final_res,
'model_attributes':'Please download your PDF here','text_paragraph':''})
self.response.out.write(html)
app_loading = webapp.WSGIApplication([('/.*', pdfPage_loading)], debug=True)
app_done = webapp.WSGIApplication([('/.*', pdfPage_done)], debug=True)
def main():
##Here is the problematic part:
if get_jid(pdf_t, pdf_nop, pdf_p)!='done':
run_wsgi_app(app_pre)
else:
run_wsgi_app(app)
if __name__ == '__main__':
main()
First off, you do not need multiple WSGIApplication handlers. You can base requests off of URLs and GET/POST params.
I would use a mixture of tasks and the Channel API:
When a user visits the page:
Create a channel token for the user to use
Make the REST API calls
Kick off a task (described below) passing it the created token and the id of the PiCloud job
Show the "loading page" and have the user connect to the Channel API with the token created in step 1
In the task, have it check on the status of the PiCloud job. If the status is not done, have the task setup a new task to call itself again, and check on the status of the job. Once the job is complete, pass the data along through the Channel API to the user so that they can then load the contents of the page or redirect to a page that will have the contents ready for them.
Example status check code (this is using the example code from PiCloud, you can use your own requests as you have been to get the status through urlfetch - this is merely meant as an example so you can plug your code in to work as needed):
import cloud
from google.appengine.api import taskqueue
from google.appengine.api import channel
#<Your other hanlders here>
class CheckStatus(webapp.RequestHandler):
def post(self):
job_id = self.request.get('job_id')
token = self.request.get('token')
status = cloud.get('status')
if status != 'done':
taskqueue.add(url='/checkstatus', params={'job_id': job_id, 'token': token}, method="POST", countdown=3)
return
channel.send_message(token, '<some message here>')
app = webapp.WSGIApplication([('/done', pdfPage_done),
('/checkstatus', CheckStatus),
('/.*', pdfPage_loading)], debug=True)
def main():
run_wsgi_app(app)
if __name__ == '__main__':
main()
I am struggling getting a Rest API Post to work with a vendor api and hope someone can give me a pointer.
The intent is to feed a cli command to the post body and pass to a device which returns the output.
The call looks like this : ( this works for all other calls but this is different because of posting to body)
def __init__(self,host,username,password,sid,method,http_meth):
self.host=host
self.username= username
self.password= password
self.sid=sid
self.method=method
self.http_meth=http_meth
def __str__(self):
self.url = 'http://' + self.host + '/rest/'
self.authparams = urllib.urlencode({ "session_id":self.sid,"method": self.method,"username": self.username,
"password": self.password,
})
call = urllib2.urlopen(self.url.__str__(), self.authparams).read()
return (call)
No matter how I have tried this I cant get it to work properly. Here is an excerpt from the API docs that explains how to use this method:
To process these APIs, place your CLI commands in the HTTP post buffer, and then place the
method name, session ID, and other parameters in the URL.
Can anyone give me an idea of how to properly do this. I am not a developer and am trying to learn this correctly. For example if I wanted to send the command "help" in the post body?
Thanks for any guidance
Ok this was ridiculously simple and I was over-thinking this. I find that I can sometimes look at a much higher level than a problem really is and waist time. Anyway this is how it should work:
def cli(self,method):
self.url = ("http://" + str(self.host) + "/rest//?method=" +str(method)+ "&username=" +str(self.username)+ "&password=" +str(self.password)+ "&enable_password=test&session_id="+str(self.session_id))
data="show ver"
try:
req = urllib2.Request(self.url)
response = urllib2.urlopen(req,data)
result = response.read()
print result
except urllib2.URLError, e:
print e.reason
The cli commands are just placed in the buffer and not encoded....