I am developping a python flask app to launch databricks jobs (executing IPython notebooks) without using databricks website.
I have a post request lauching the execution of a notebook.
#app.route('/', methods=['GET', 'POST'])
def main():
response = requests.get(
url=url1,
json=data,
headers={'Authorization': 'Bearer %s' % TOKEN }
)
notebookDict[item] = json.loads(response.text)
# [... some logic to sort notebook dict ...]
for item in listeNomNotebook:
listeNomNotebook[item].sort()
# my post method for executing notebook through api
if request.method == 'POST':
#[... some variable ...]
params = {
'run_name' : f'{notebook_name}',
'existing_cluster_id' : cluster_id,
'notebook_task' : {
'notebook_path' : f'{pathstr}',
'base_parameters' : parameters
},
}
response = requests.post(
url=url2,
json=params,
headers={'Authorization': 'Bearer %s' % TOKEN }
)
return render_template('index.html', data=listeNomNotebook)
What i want is, from the response of this POST request, which is a job_id, to execute a GET request with this job_id as a parameters to know the result of the execution, failure or success.
I want then to update the website dinamically to show the lifecycle of the job. By this I mean having my GET request in a loop until the state from the response shows terminated. Here is my test code to do this :
job_id = response.json()
print(job_id)
state = ""
while True:
result = requests.get(
url= url3,
json=job_id,
headers={'Authorization': 'Bearer %s' % TOKEN }
)
dict_result = result.json()
life_cycle = dict_result["state"]["life_cycle_state"]
state = dict_result["state"]["state_message"]
print(life_cycle)
if life_cycle == "TERMINATED":
state = dict_result["state"]["result_state"]
print(state)
break
sleep(10)
I tryed to use ajax but the CORS policy wouldn't let me use that solution (even with an add on on my browser and using flask-cors).
How can i show the result of my notebook on my index page dynamically ? Showing the lifecycle : pending, running , terminated.
And then showing the result : Success or Failure.
All of this is part of the response from the GET request of the databricks api :
api/2.0/jobs/runs/get
I suggest you do it in this order:
Post to / to start the request, get the job_id in the frontend.
In the front-end, periodically query for job status.
In case the problem is with CORS, then please open a question concerning that after verifying that existing answers don't work. There could be many reasons why you get CORS errors, be sure to state the actual blocker.
Related
I have used the following guide - https://developer.ebay.com/marketplace-account-deletion
So I've created a flask based website on python and uploaded in Heroku,
https://ebay-deletion.herokuapp.com/
If you create a query parameter, for the challenge code like so:
https://ebay-deletion.herokuapp.com/?challenge_code=123
It returns a challenge response and doc type (JSON).
I don't understand but why but I get this error below:
Notification delivery failed with HTTP status code 405 from https://ebay-deletion.herokuapp.com. Please ensure that the marketplace account deletion notification endpoint is ready to receive notifications.
Any thoughts on how to solve this?
Ebay's help page is just so terrible. Their python code examples for hashing in the top link have semi-colons after each line! It's close to completely useless.
Here is the Python Flask code:
#app.route('/')
def index():
args = request.args
args_dict = args.to_dict()
try:
resp_hash = hashlib.sha256(
args_dict['challenge_code'].encode() + verification_token.encode() + endpoint.encode())
resp = {'challengeResponse': resp_hash.hexdigest()}
return jsonify(resp), 200, {'content-type': 'application/json'}
except KeyError:
err = {'status_code': 400, 'message': 'no challenge response in params'}
return jsonify(err), 400, {'content-type': 'application/json'}
When I try to access the header of a request in a function adnotated with #app.before_request, it is always None. Anyone had this problem before?
Here is the function:
#app.before_request
def verifyToken():
if request.endpoint in ['myEndpoint']
auth = request.headers.get('Authorization')
if auth.startswith('Bearer '):
jwtToken = auth[7:]
try:
decoded = jwt.decode(jwtToken, 'secret_key', algorithms=['HS256'])
except jwt.ExpiredSignatureError as e:
responseObject = {
"status": "failed",
"message": "Token has expired. Please login again."
}
return jsonify(responseObject), 401
I was being 5 months in Flask, I discover Flask Embeded Middleware fires up 2 times, the first one got None value and the second one is the client request and each fire up has different request.method, the OPTIONS and <DEFAULT METHOD> of the client, why there is an OPTIONS method first in before...request and after...request befoe goes in client request?, I'm still searching for reference
and that is why I just catch it through conditional statement if request.method != 'OPTIONS': and here how it is look like
from flask import request
def my_before_request():
if request.method !== 'OPTIONS'
token = request.headers['<header var>']
# do something...
# same as after request
The OPTIONS method could have been triggered by the browser due to request being preflighted. This is done to determine if the actual request is safe to send. Refer CORS
I lately started using Flask in one of my projects to provide data via a simple route. So far I return a json file containing the data and some other information. When running my Flask app I see the status code of this request in terminal. I would like to return the status code as a part of my final json file. Is it possible to catch the same code I see in terminal?
Some simple might look like this
from flask import Flask
from flask import jsonify
app = Flask(__name__)
#app.route('/test/<int1>/<int2>/')
def test(int1,int2):
int_sum = int1 + int2
return jsonify({"result":int_sum})
if __name__ == '__main__':
app.run(port=8082)
And in terminal I get:
You are who set the response code (by default 200 on success response), you can't catch this value before the response is emited. But if you know the result of your operation you can put it on the final json.
#app.route('/test/<int1>/<int2>/')
def test(int1, int2):
int_sum = int1 + int2
response_data = {
"result": int_sum,
"sucess": True,
"status_code": 200
}
# make sure the status_code on your json and on the return match.
return jsonify(response_data), 200 # <- the status_code displayed code on console
By the way if you access this endpoint from a request library, on the response object you can find the status_code and all the http refered data plus the json you need.
Python requests library example
import requests
req = requests.get('your.domain/test/3/3')
print req.url # your.domain/test/3/3
print req.status_code # 200
print req.json() # {u'result': 6, u'status_code: 200, u'success': True}
You can send HTTP status code as follow:
#app.route('/test')
def test():
status_code = 200
return jsonify({'name': 'Nabin Khadka'}, status_code) # Notice second element of the return tuple(return)
This way you can control what status code to return to the client (typically to web browser.)
I am new to backend development using Flask and I am getting stuck on a confusing problem. I am trying to send data to an endpoint whose Timeout session is 3000 ms. My code for the server is as follows.
from flask import Flask, request
from gitStat import getGitStat
import requests
app = Flask(__name__)
#app.route('/', methods=['POST', 'GET'])
def handle_data():
params = request.args["text"].split(" ")
user_repo_path = "https://api.github.com/users/{}/repos".format(params[0])
first_response = requests.get(
user_repo_path, auth=(
'Your Github Username', 'Your Github Password'))
repo_commits_path = "https://api.github.com/repos/{}/{}/commits".format(params[
0], params[1])
second_response = requests.get(
repo_commits_path, auth=(
'Your Github Username', 'Your Github Password'))
if(first_response.status_code == 200 and params[2] < params[3] and second_response.status_code == 200):
values = getGitStat(params[0], params[1], params[2], params[3])
response_url = request.args["response_url"]
payload = {
"response_type": "in_channel",
"text": "Github Repo Commits Status",
"attachments": [
{
"text": values
}
]
}
headers = {'Content-Type': 'application/json',
'User-Agent': 'Mozilla /5.0 (Compatible MSIE 9.0;Windows NT 6.1;WOW64; Trident/5.0)'}
response = requests.post(response_url, json = test, headers = headers)
else:
return "Please enter correct details. Check if the username or reponame exists, and/or Starting date < End date. \
Also, date format should be MM-DD"
My server code takes arguments from the request it receives and from that request's JSON object, it extracts parameters for the code. This code executes getGitStats function and sends the JSON payload as defined in the server code to the endpoint it received the request from.
My problem is that I need to send a text confirmation to the endpoint that I have received the request and data will be coming soon. The problem here is that the function, getGitStats take more than a minute to fetch and parse data from Github API.
I searched the internet and found that I need to make this call asynchronous and I can do that using queues. I tried to understand the application using RQ and RabbitMQ but I neither understood nor I was able to convert my code to an asynchronous format. Can somebody give me pointers or any idea on how can I achieve this?
Thank you.
------------Update------------
Threading was able to solve this problem. Create another thread and call the function in that thread.
If you are trying to have a async task in request, you have to decide whether you want the result/progress or not.
You don't care about the result of the task or if there where any errors while processing the task. You can just process this in a Thread and forget about the result.
If you just want to know about success/fail for the task. You can store the state of the task in Database and query it when needed.
If you want progress of the tasks like (20% done ... 40% done). You have to use something more sophisticated like celery, rabbitMQ.
For you i think option #2 fits better. You can create a simple table GitTasks.
GitTasks
------------------------
Id(PK) | Status | Result
------------------------
1 |Processing| -
2 | Done | <your result>
3 | Error | <error details>
You have to create a simple Threaded object in python to processing.
import threading
class AsyncGitTask(threading.Thread):
def __init__(self, task_id, params):
self.task_id = task_id
self.params = params
def run():
## Do processing
## store the result in table for id = self.task_id
You have to create another endpoint to query the status of you task.
#app.route('/TaskStatus/<int:task_id>')
def task_status(task_id):
## query GitTask table and accordingly return data
Now that we have build all the components we have to put them together in your main request.
from Flask import url_for
#app.route('/', methods=['POST', 'GET'])
def handle_data():
.....
## create a new row in GitTasks table, and use its PK(id) as task_id
task_id = create_new_task_row()
async_task = AsyncGitTask(task_id=task_id, params=params)
async_task.start()
task_status_url = url_for('task_status', task_id=task_id)
## This is request you can return text saying
## that "Your task is being processed. To see the progress
## go to <task_status_url>"
currently I am trying to migrate a working php paypal payflow implementation to a new python-based system.
I use secure token together with hosted checkout pages. The secure token works fine and I get redirected to the checkout page as well (although it has horrible formatting errors).
THE PROBLEM: after the payment it should redirect to the return url. This works BUT 'processTransaction.do' is appended to it. So my return url is defined as:
'https://mywebsite.com/paypal/succes/'
but i get redirected to
'https://mywebsite.com/paypal/succes/processTransaction.do'
and this raises a 404.
My secure token request parameters:
params = {}
params["PARTNER"] = "paypal"
params["VENDOR"] = "...."
params["TRXTYPE"] = "S"
params["AMT"] = payment_amount #amount to pay
params["CREATESECURETOKEN"] = "Y"
params["SECURETOKENID"] = time.time() #needs to be unique
params["USER"] = "...."
params["PWD"] = "...."
Then I send the request and catch the return which looks like this:
RESULT=0&SECURETOKEN=QQQc0rQZ8TkKSNMqU3Mg2og7o
SECURETOKENID=1431563231.24&RESPMSG=Approved
Afterwards I send the request for the checkout page with the following paramters:
params["SECURETOKEN"] = securetoken
params["SECURETOKENID"] = securetokenid
to: https://payflowpro.paypal.com
I use this code to send the requests:
data = urllib.urlencode(params)
request = urllib2.Request(url, data)
response = urllib2.urlopen(request)
response_text = response.read()
The return url is set in the paypal manager with return type as POST and "Show Confirmation Page" is set to "On my website".
Does somebody know what is wrong and how to fix it?
Thanks!