On GAE, I need to make some REST calls to a PiCloud server, then create an output page based on PiCloud return values. However, it will take several minutes for PiCloud to process the model. Thus I am wondering if I could create a 'loading' page first, and after finishing calculation, present the real output page.
In detail, the questions is how do I keep checking the status of my REST service and then generate different HTML pages based upon.
I appreciate any suggestions and comments!
PS: jQuery BlockUI seems to be good example, but it requires to estimate the timeout duration, which I could not guess...
Function to Call REST Service:
def get_jid(pdf_t, pdf_nop, pdf_p):
response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers)
jid= json.loads(response.content)['jid']
output_st = "running"
while output_st!="done":
response_st = urlfetch.fetch(url='https://api.picloud.com/job/?jids=%s&field=status' %jid, headers=http_headers)
output_st = json.loads(response_st.content)['info']['%s' %jid]['status']
url_val = 'https://api.picloud.com/job/result/?jid='+str(jid)
response_val = urlfetch.fetch(url=url_val, method=urlfetch.GET, headers=http_headers)
output_val = json.loads(response_val.content)['result']
return(jid, output_st, output_val)
Generate HTML Page:
class pdfPage_loading(webapp.RequestHandler):
def post(self):
final_res=get_jid(pdf_t, pdf_nop, pdf_p)[2]
html = html + template.render(templatepath + 'popup_pdf_eco.html', {
'title':'Ubertool',
'model_page':'',
'model_attributes':'Please wait','text_paragraph':''})
self.response.out.write(html)
class pdfPage_done(webapp.RequestHandler):
def post(self):
final_res=get_jid(pdf_t, pdf_nop, pdf_p)[2]
html = html + template.render(templatepath + 'popup_pdf_eco.html', {
'title':'Ubertool',
'model_page':final_res,
'model_attributes':'Please download your PDF here','text_paragraph':''})
self.response.out.write(html)
app_loading = webapp.WSGIApplication([('/.*', pdfPage_loading)], debug=True)
app_done = webapp.WSGIApplication([('/.*', pdfPage_done)], debug=True)
def main():
##Here is the problematic part:
if get_jid(pdf_t, pdf_nop, pdf_p)!='done':
run_wsgi_app(app_pre)
else:
run_wsgi_app(app)
if __name__ == '__main__':
main()
First off, you do not need multiple WSGIApplication handlers. You can base requests off of URLs and GET/POST params.
I would use a mixture of tasks and the Channel API:
When a user visits the page:
Create a channel token for the user to use
Make the REST API calls
Kick off a task (described below) passing it the created token and the id of the PiCloud job
Show the "loading page" and have the user connect to the Channel API with the token created in step 1
In the task, have it check on the status of the PiCloud job. If the status is not done, have the task setup a new task to call itself again, and check on the status of the job. Once the job is complete, pass the data along through the Channel API to the user so that they can then load the contents of the page or redirect to a page that will have the contents ready for them.
Example status check code (this is using the example code from PiCloud, you can use your own requests as you have been to get the status through urlfetch - this is merely meant as an example so you can plug your code in to work as needed):
import cloud
from google.appengine.api import taskqueue
from google.appengine.api import channel
#<Your other hanlders here>
class CheckStatus(webapp.RequestHandler):
def post(self):
job_id = self.request.get('job_id')
token = self.request.get('token')
status = cloud.get('status')
if status != 'done':
taskqueue.add(url='/checkstatus', params={'job_id': job_id, 'token': token}, method="POST", countdown=3)
return
channel.send_message(token, '<some message here>')
app = webapp.WSGIApplication([('/done', pdfPage_done),
('/checkstatus', CheckStatus),
('/.*', pdfPage_loading)], debug=True)
def main():
run_wsgi_app(app)
if __name__ == '__main__':
main()
Related
So, in my main app.py, I have :
app = create_app()
...
app.run()
In __init__.py I have:
def create_app():
...
app = Flask(__name__)
setup_logging(app)
return app
def setup_logging(app):
RequestID(app)
handler = logging.FileHandler(app.container.config.get("app.LOG_FILE"))
handler.setFormatter(logging.Formatter("%(asctime)s : %(levelname)s : %(request_id)s - %(message)s"))
handler.addFilter(RequestIDLogFilter()) # << Add request id contextual filter
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(level="DEBUG")
And in my routes.py, I have: (This wiring/linking is done with the help of dependency injection)
def configure(app):
app.add_url_rule("/", "check", check)
...
def check():
logging.info("You hit /")
# Here I want to return the Request UUID that's generated by RequestLogIDFilter
return make_response(jsonify(dict(ok=True)), 200)
How do I access the Request UUID generated by the RequestLogIDFilter? In my log file I correctly see the log messages as:
2022-08-15 07:00:18,030 : INFO : 27b437fd-98be-4bc9-a609-912043e3a38e - Log message test memberId=9876
I want to take this value 27b437fd-98be-4bc9-a609-912043e3a38e and include it in the response headers as X-REQUEST-UUID: 27b437fd-98be-4bc9-a609-912043e3a38e
An alternative was to rip out RequestLogIDFilter and only work with flask-request-id-header except, there's no way to write it to log file (logging.FileHandler does not have a write() method so it fails)
This is something trivial I'm sure but the official docs nowhere mention how to access these request ids for logging to file or sending back as a response header.
If you are using flask_log_request_id then you probably want this as given in their github repo
Code copied here in case link does not work
from flask_log_request_id import current_request_id
#app.after_request
def append_request_id(response):
response.headers.add('X-REQUEST-ID', current_request_id())
return response
Using Tornado, I have a POST request that takes a long time as it makes many requests to another API service and processes the data. This can take minutes to fully complete. I don't want this to block the entire web server from responding to other requests, which it currently does.
I looked at multiple threads here on SO, but they are often 8 years old and the code does not work anylonger as tornado removed the "engine" component from tornado.gen.
Is there an easy way to kick off this long get call and not have it block the entire web server in the process? Is there anything I can put in the code to say.. "submit the POST response and work on this one function without blocking any concurrent server requests from getting an immediate response"?
Example:
main.py
def make_app():
return tornado.web.Application([
(r"/v1", MainHandler),
(r"/v1/addfile", AddHandler, dict(folderpaths = folderpaths)),
(r"/v1/getfiles", GetHandler, dict(folderpaths = folderpaths)),
(r"/v1/getfile", GetFileHandler, dict(folderpaths = folderpaths)),
])
if __name__ == "__main__":
app = make_app()
sockets = tornado.netutil.bind_sockets(8888)
tornado.process.fork_processes(0)
tornado.process.task_id()
server = tornado.httpserver.HTTPServer(app)
server.add_sockets(sockets)
tornado.ioloop.IOLoop.current().start()
addHandler.py
class AddHandler(tornado.web.RequestHandler):
def initialize(self, folderpaths):
self.folderpaths = folderpaths
def blockingFunction(self):
time.sleep(320)
post("AWAKE")
def post(self):
user = self.get_argument('user')
folderpath = self.get_argument('inpath')
outpath = self.get_argument('outpath')
workflow_value = self.get_argument('workflow')
status_code, status_text = validateInFolder(folderpath)
if (status_code == 200):
logging.info("Status Code 200")
result = self.folderpaths.add_file(user, folderpath, outpath, workflow_value)
self.write(result)
self.finish()
#At this point the path is validated.
#POST response should be send out. Internal process should continue, new
#requests should not be blocked
self.blockingFunction()
Idea is that if input-parameters are validated the POST response should be sent out.
Then internal process (blockingFunction()) should be started, that should not block the Tornado Server from processing another API POST request.
I tried defining the (blockingFunction()) as async, which allows me to process multiple concurrent user requests - however there was a warning about missing "await" with async method.
Any help welcome. Thank you
class AddHandler(tornado.web.RequestHandler):
def initialize(self, folderpaths):
self.folderpaths = folderpaths
def blockingFunction(self):
time.sleep(320)
post("AWAKE")
async def post(self):
user = self.get_argument('user')
folderpath = self.get_argument('inpath')
outpath = self.get_argument('outpath')
workflow_value = self.get_argument('workflow')
status_code, status_text = validateInFolder(folderpath)
if (status_code == 200):
logging.info("Status Code 200")
result = self.folderpaths.add_file(user, folderpath, outpath, workflow_value)
self.write(result)
self.finish()
#At this point the path is validated.
#POST response should be send out. Internal process should continue, new
#requests should not be blocked
await loop.run_in_executor(None, self.blockingFunction)
#if this had multiple parameters it would be
#await loop.run_in_executor(None, self.blockingFunction, param1, param2)
Thank you #xyres
Further read: https://www.tornadoweb.org/en/stable/faq.html
I have a function that generates all video URLs of some user on Twitch
def get_videos(cursor=None): # functiont to retrieve all vod URLs possible, kinda slow for now
params_get_videos = {('user_id', userid_var)} # params for request.get
if cursor is not None: # check if there was a cursor value passed
params_get_videos = list(params_get_videos) + list({('after', cursor)}) # add another param for pagination
url_get_videos = 'https://api.twitch.tv/helix/videos' # URL to request data
response_get_videos = session.get(url_get_videos, params=params_get_videos, headers=headers) # get the data
reponse_get_videos_json = response_get_videos.json() # parse and interpret data
file = open(MyUsername+" videos.txt", "w")
for i in range(0, len(reponse_get_videos_json['data'])): # parse and interpret data
file.write(reponse_get_videos_json['data'][i]['url'] +'\n') # parse and interpret data
if 'cursor' in reponse_get_videos_json['pagination']: # check if there are more pages
get_videos(reponse_get_videos_json['pagination']['cursor']) # iterate the function until there are no more pages
this works perfectly fine on its own (with other functions) but whenever i try to call it from a dummy flask server like this
from flask import Flask
from flask import render_template
from flask import request
from main import *
app = Flask(__name__)
#app.route('/')
def hello_world():
return render_template("hello.html")
#app.route('/magic', methods=['POST', 'GET'])
def get_username():
username = request.form.get('username')
get_videos()
return ("Success")
It no longer acts recursive and only prints first 20 values. What am I doing wrong?
I'm new also so I don't have enough reputation to make a comment, I have to post an answer. I'm confused by your get_username method, once you grab the username from the form you're not sending it anywhere? It looks like on your get_videos method you are maybe hard coding the username by saving a variable called MyUsername outside of this method?
What you should do imo is send the username you grab from the form, do your get_videos method like this.
get_videos(username)
And change your other method to this
def get_videos(MyUsername, cursor=None):
I am new to backend development using Flask and I am getting stuck on a confusing problem. I am trying to send data to an endpoint whose Timeout session is 3000 ms. My code for the server is as follows.
from flask import Flask, request
from gitStat import getGitStat
import requests
app = Flask(__name__)
#app.route('/', methods=['POST', 'GET'])
def handle_data():
params = request.args["text"].split(" ")
user_repo_path = "https://api.github.com/users/{}/repos".format(params[0])
first_response = requests.get(
user_repo_path, auth=(
'Your Github Username', 'Your Github Password'))
repo_commits_path = "https://api.github.com/repos/{}/{}/commits".format(params[
0], params[1])
second_response = requests.get(
repo_commits_path, auth=(
'Your Github Username', 'Your Github Password'))
if(first_response.status_code == 200 and params[2] < params[3] and second_response.status_code == 200):
values = getGitStat(params[0], params[1], params[2], params[3])
response_url = request.args["response_url"]
payload = {
"response_type": "in_channel",
"text": "Github Repo Commits Status",
"attachments": [
{
"text": values
}
]
}
headers = {'Content-Type': 'application/json',
'User-Agent': 'Mozilla /5.0 (Compatible MSIE 9.0;Windows NT 6.1;WOW64; Trident/5.0)'}
response = requests.post(response_url, json = test, headers = headers)
else:
return "Please enter correct details. Check if the username or reponame exists, and/or Starting date < End date. \
Also, date format should be MM-DD"
My server code takes arguments from the request it receives and from that request's JSON object, it extracts parameters for the code. This code executes getGitStats function and sends the JSON payload as defined in the server code to the endpoint it received the request from.
My problem is that I need to send a text confirmation to the endpoint that I have received the request and data will be coming soon. The problem here is that the function, getGitStats take more than a minute to fetch and parse data from Github API.
I searched the internet and found that I need to make this call asynchronous and I can do that using queues. I tried to understand the application using RQ and RabbitMQ but I neither understood nor I was able to convert my code to an asynchronous format. Can somebody give me pointers or any idea on how can I achieve this?
Thank you.
------------Update------------
Threading was able to solve this problem. Create another thread and call the function in that thread.
If you are trying to have a async task in request, you have to decide whether you want the result/progress or not.
You don't care about the result of the task or if there where any errors while processing the task. You can just process this in a Thread and forget about the result.
If you just want to know about success/fail for the task. You can store the state of the task in Database and query it when needed.
If you want progress of the tasks like (20% done ... 40% done). You have to use something more sophisticated like celery, rabbitMQ.
For you i think option #2 fits better. You can create a simple table GitTasks.
GitTasks
------------------------
Id(PK) | Status | Result
------------------------
1 |Processing| -
2 | Done | <your result>
3 | Error | <error details>
You have to create a simple Threaded object in python to processing.
import threading
class AsyncGitTask(threading.Thread):
def __init__(self, task_id, params):
self.task_id = task_id
self.params = params
def run():
## Do processing
## store the result in table for id = self.task_id
You have to create another endpoint to query the status of you task.
#app.route('/TaskStatus/<int:task_id>')
def task_status(task_id):
## query GitTask table and accordingly return data
Now that we have build all the components we have to put them together in your main request.
from Flask import url_for
#app.route('/', methods=['POST', 'GET'])
def handle_data():
.....
## create a new row in GitTasks table, and use its PK(id) as task_id
task_id = create_new_task_row()
async_task = AsyncGitTask(task_id=task_id, params=params)
async_task.start()
task_status_url = url_for('task_status', task_id=task_id)
## This is request you can return text saying
## that "Your task is being processed. To see the progress
## go to <task_status_url>"
I'm trying to subscribe to a tag. It appears that the callback URL is being called correctly with a hub.challenge and hub.mode, and I figured out how to access the challenge using self.request.get('hub.challenge'). I thought I was just supposed to echo the challenge, but that doesn't appear to work since I receive the following errors in the GAE logs:
InstagramAPIError: (400) APISubscriptionError-Challenge verification failed. Sent "647bf6dbed31465093ee970577ce1b72", received "
647bf6dbed31465093ee970577ce1b72
".
Here is the full handler:
class InstagramHandler(BaseHandler):
def get(self):
def process_tag_update(update):
update = update
mode = self.request.get('hub.mode')
challenge = self.request.get('hub.challenge')
verify_token = self.request.get('hub.verify_token')
if challenge:
template_values = {'challenge':challenge}
path = os.path.join(os.path.dirname(__file__), '../templates/instagram.html')
html = template.render(path, template_values)
self.response.out.write(html)
else:
reactor = subscriptions.SubscriptionsReactor()
reactor.register_callback(subscriptions.SubscriptionType.TAG, process_tag_update)
x_hub_signature = self.request.headers.get('X-Hub-Signature')
raw_response = self.request.data
try:
reactor.process('INSTAGRAM_SECRET', raw_response, x_hub_signature)
except subscriptions.SubscriptionVerifyError:
logging.error('Instagram signature mismatch')
So returning it as a string worked. I should have payed closer attention to the error message, but it took a helpful person on the Python IRC to point out the extra line breaks in the message. Once I put the template files on one line, it seemed to work. I can now confirm that my app is authorized via Instagram's list subscription URL.