I have 2 machines A and B and A can send restful request to B as follows:
curl -XPOST -H "Content-type: application/json" -d '{"data":"python /tmp/demo.py","action":"demo"}' 'http://192.168.95.8:51888/api/host'
I have deployed an api service on B and when such request is received, B will execute the python script /tmp/demo.py and the execution may last 0.5-3 hours.
My question is:
1) How to write a job on A that keeps tracking the status of the task running on B and end it self when the task finishes successfully or failed?
2) In the tracking job, how to add a module that can kill itself after exceeding a pre-set time threshold?
Treat the job as an HTTP resource. When you do POST /api/host, that request creates a new id for that job and returns it. For good use of HTTP, the response would contain a Location header with the URL of the resource where the job's status can be checked, e.g.:
POST /api/hosts
Content-type: application/json
{"data":"python /tmp/demo.py","action":"demo"}
HTTP/1.1 201 Created
Location: /api/host/jobs/c2de232b-f63e-4178-a053-d3f3459ab538
You can now GET /api/host/jobs/c2de232b-f63e-4178-a053-d3f3459ab538 at any time and see what status the job has, e.g.:
{"status": "pending"}
You may POST commands to that resource, e.g. for cancelling it.
How exactly your HTTP API would get the status of that Python script is obviously up to you. Perhaps it can communicate with it over a socket, or the job itself will periodically write its status to some database or file.
Related
I am using airflow to schedule and automate Python scripts housed on a Ubuntu server. The DAG triggers a CURL request that hits a Flask API on the same machine which actually runs the script. Here is a high level overview of the flow:
Airflow --> Curl request --> Flask API --> Python Script
DAG Task:
t2 = BashOperator (
task_id='extract_pcty_data',
bash_command=f"""curl -d '{dataset}' -H 'Content-Type: application/json' -X POST {base_url}{endpoint}""",
)
Endpoint Registration:
api.add_resource(paylocity, "/api/v1/application/paylocity")
Resource Object:
class paylocity(Resource):
def __init__(self):
self.reqparse = reqparse.RequestParser()
def get(self):
return 200
def post(self):
try:
if request.json:
data = request.json
query = data['dataset']
pcty = PaylocityAPI()
pcty.auth()
pcty.get_employees()
pcty.get_paystatements()
pcty.load_dataset()
pcty.clean_up()
return 200
except Exception as e:
print(traceback.print_exc(e))
raise ValueError(e)
The issue I am running into, is that the script will fail for some reason which gets caught by the try/catch block and then raises the value error - but it does not cause the script to fail because the HTTP request response returned is 500 - Internal Server Error. What I am looking for is a simple and elegant way to interpret an HTTP response that is not 200 - OK as a "failure" and raising something like a ValueError or AirflowException to cause the task to fail. Any guidance or support would be greatly appreciated!
To those of you from Google looking for a simple and elegant answer to this or a similar question. Curl has a few flags that allow you to specify how you want the fail behavior of a request to act. For my specific scenario: --fail was the most appropriate. There is also --fail-with-body that allows you to get the content of the fail response rather than just the non-zero exit code. From their docs:
-f, --fail
(HTTP) Fail fast with no output at all on server errors. This is useful to enable scripts and users to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.
This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).
Example:
curl --fail https://example.com
I have a code in python:
cmd = "curl -d 'protection={protection}&Code={Code}' -X POST https://example.com/web/services/toaf6.php"
os.system(cmd.format(protection=protection, Code=Code))
I want to transform this so it uses pythons request libraries???
This request should be sync, but endpoint will return immediately response and I will store transaction ID so I can ask some other endpoint for job status.
I have an API, which performs the task likes this:
Client sends the request.
The task is received by the celery worker and put in the message queue.
The response is sent as soon as the request is received and the task is executed.
Now, I am load testing my application on JMeter. In my server logs, whenever I send a request to the server, the response is received asap and the task is executed. But, JMeter is executing the whole query and sending me the response after the entire query is executed. Now, I want JMeter to send me the response as soon as the request is received, not when the whole query is executed.I want to record the response times of the server, when mulitple queries are sent.
How can I achieve this? Can anybody help me with this?
Thanks
I'm facing some difficulties executing GET request with access token via python.
For some reason when I execute the request from POSTMAN I get the expected result, however, when using python I get connection error :
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
I believe that my script is being blocked by the server to prevent scrapping attempts, but I'm not sure.
This is what I'm running via POSTMAN :
https://www.operation.io/web/alerts?access_token=6ef06fee265de1fa640b6a444ba47--z&envId=58739be2c25e76a202c9dd3e&folderId=active&sortBy=status
And this is what I'm running from inside the script :
response = requests.get('https://www.operation.io/web/alerts', headers={'Authorization': 'access_token 6ef06fee265de1fa640b6a444ba47--z'})
Please notice that the url and the access token are not real, so please don't try to use.
How can I make it work via the script and not only via postman?
Thank you for your help/
It's hard to tell without actually being able to test the solutions, but I would suggest 2 hacks that worked for me in the past:
Change "Authorization" to "authorization"
Change "6ef06fee265de1fa640b6a444ba47--z" to "Token 6ef06fee265de1fa640b6a444ba47--z" (add a space as well)
Put it all together:
response = requests.get('https://www.operation.io/web/alerts', headers={'authorization': 'Token 6ef06fee265de1fa640b6a444ba47--z'})
Since via postman you're sending the token as query param(as opposed to header), you can try this :
response = requests.get('https://www.operation.io/web/alerts', params={'access_token': '6ef06fee265de1fa640b6a444ba47--z'})
Assuming the API you're using accepts it as query param rather than as headers (which is my guess from the postman request)
I receive posted data and immediately return an empty 200 OK response. After that I will process the received data. I'm considering how to do it with a teardown function but I didn't find how to pass it the received data:
#app.route('/f', methods = ['POST'])
def f():
data = request.stream.read()
return ''
#app.teardown_request
def teardwon_request(exception=None):
# How to use posted data here?
Flask version is 0.10.1
I'm trying to implement a Paypal IPN listener
https://developer.paypal.com/webapps/developer/docs/classic/ipn/gs_IPN/#overview
Notice that the listener's HTTP 200 response happens before the listener's IPN message.
You are overcomplicating things; just send a request from your Flask server in the request handler. Paypal IPN notifications just require a empty 200 response, Paypal does not mandate that you send the 200 OK before you can send the HTTP request to their servers.
The overview page is indeed confusing, but the PHP code posted won't close the request until the Paypal IPN post back to their server has completed either.
If this was a hard requirement (making this a terrible design), you'd have to handle the request back to Paypal asynchronously. You can do this with a separate thread, for example, using a queue, push in the data you received from the IPN, and have a separate thread poll the queue and communicate to Paypal from that thread. Or you could use Celery to simplify the job (push a task out to be handled asynchronously). Either way, this would let you close the incoming request early.