I've been trying to set up a subscription for my app with the Realtime Updates API but there have been some issues. For starters, this is the error I keep getting:
{"error":{"message":"(#2200) callback verification failed: Operation timed out after 6000 milliseconds with 0 bytes received","type":"OAuthException","code":2200}}
I've followed the documentation appropriately and configured a Flask endpoint on an Amazon EC2 instance that handles HTTP GET and POST. What happens is I hit and endpoint myself, manually, to invoke the subscription code.
curl -i -X GET http://public-ip-of-ec2:5000/subscribe
The above curl calls a script running in a flask application at the route of /subscribe on my ec2 instance. To make the POST with the required query string parameters including our access_token, object, fields, verify_token, and callback_url I'm using the python HTTP library requests.
VERIFY_TOKEN = 'my_verify_token'
#app.route('/subscribe')
def subscribe():
global VERIFY_TOKEN
FB_CLIENT_ID = 'my_app_id'
# access_token is sent as a query string parameter
APP_ACCESS_TOKEN = 'my_app_access_token'
# object, fields, callback_url, and verify_token are sent as urllib.urlencode([('param','val')])
CALLBACK_URL = 'http://my-public-ec2-ip:5000/'
payload_url = "https://graph.facebook.com/{0}/subscriptions".format(FB_CLIENT_ID)
payload = {"access_token": APP_ACCESS_TOKEN, "object": "user", "fields": "feed", "verify_token": VERIFY_TOKEN, "callback_url": CALLBACK_URL}
r = requests.post(payload_url, data=payload)
return r.text
#app.route('/', methods=['GET','POST'])
def handle_requests():
global VERIFY_TOKEN
if request.method == 'GET':
mode = request.args.get('hub.mode')
challenge = request.args.get('hub.challenge')
verification = request.args.get('hub.verify_token')
# if we have our verification token back echo the challenge back to facebook
if verification == VERIFY_TOKEN:
return challenge
elif request.method == 'POST':
# do some stuff with the updates
I'm confused as to why I'm getting
{"error":{"message":"(#2200) callback verification failed: Operation timed out after 6000 milliseconds with 0 bytes received","type":"OAuthException","code":2200}}
because when I fire up my flask application I can see a GET request from 173.252.110.113, which is a Facebook IP address. I have properly tested to make sure I'm echoing back the correct data by printing the challenge to my log for testing. So the code IS returning the challenge which facebook requires to verify subscription and at that point the subscription SHOULD be successful, but the aforementioned error is what I'm getting. Could it possibly just be a security issue I need to add a permission for in ec2 security groups or something??
Thanks in advance for the help!
ANSWER:
Facebook without forewarning sends multiple requests to the endpoint in question and with the Flask development server those requests have no way of being handled, therefore the timeout. I fired up a gunicorn server with a few workers to test that theory and it proved to be true because I now have a successful subscription verification. For anyone else having this issue with flask:
$ sudo pip install gunicorn
$ which gunicorn
/usr/local/bin/gunicorn
# fire up your endpoint with a few gunicorn workers to handle the load
# facebook tests our endpoint with (we will use 4 workers on port 5000)
# my_app is your_app_name.py without the .py part
$ /usr/local/bin/gunicorn -w 4 -b my-local-ipv4-ip:5000 my_app:app
"https://graph.facebook.com/{0}/subscriptions".format(FB_CLIENT_ID)
is for getting current subscriptions by GET method
to make new subscription you can try :
"https://graph.facebook.com/{0}/".format(FB_CLIENT_ID)
by POST method --- import : is do not include subscriptions at the end url
Related
I have created and ran dags on a google-cloud-composer environment (dlkpipelinesv1 : composer-1.13.0-airflow-1.10.12). I am able to trigger these dags manually, and using the scheduler, but I am stuck when it comes to triggering them via cloud-functions that detect changes in a google-cloud-storage bucket.
Note that I had another GC-Composer environment (pipelines:composer-1.7.5-airflow-1.10.2) that used those same google cloud functions to trigger the relevant dags, and it was working.
I followed this guide to create the functions that trigger the dags. So I retrieved the following variables:
PROJECT_ID = <project_id>
CLIENT_ID = <client_id_retrieved_by_running_the_code_in_the_guide_within_my_gcp_console>
WEBSERVER_ID = <airflow_webserver_id>
DAG_NAME = <dag_to_trigger>
WEBSERVER_URL = f"https://{WEBSERVER_ID}.appspot.com/api/experimental/dags/{DAG_NAME}/dag_runs"
def file_listener(event, context):
"""Entry point of the cloud function: Triggered by a change to a Cloud Storage bucket.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
logging.info("Running the file listener process")
logging.info(f"event : {event}")
logging.info(f"context : {context}")
file = event
if file["size"] == "0" or "DTM_DATALAKE_AUDIT_COMPTAGE" not in file["name"] or ".filepart" in file["name"].lower():
logging.info("no matching file")
exit(0)
logging.info(f"File listener detected the presence of : {file['name']}.")
# id_token = authorize_iap()
# make_iap_request({"file_name": file["name"]}, id_token)
make_iap_request(url=WEBSERVER_URL, client_id=CLIENT_ID, method="POST")
def make_iap_request(url, client_id, method="GET", **kwargs):
"""Makes a request to an application protected by Identity-Aware Proxy.
Args:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
method: The request method to use
('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
Returns:
The page body, or raises an exception if the page couldn't be retrieved.
"""
# Set the default timeout, if missing
if "timeout" not in kwargs:
kwargs["timeout"] = 90
# Obtain an OpenID Connect (OIDC) token from metadata server or using service account.
open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
logging.info(f"Retrieved open id connect (bearer) token {open_id_connect_token}")
# Fetch the Identity-Aware Proxy-protected URL, including an authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(method, url, headers={"Authorization": f"Bearer {open_id_connect_token}"}, **kwargs)
if resp.status_code == 403:
raise Exception("Service account does not have permission to access the IAP-protected application.")
elif resp.status_code != 200:
raise Exception(f"Bad response from application: {resp.status_code} / {resp.headers} / {resp.text}")
else:
logging.info(f"Response status - {resp.status_code}")
return resp.json
This is the code that runs in the GC-functions
I have checked the environment details in dlkpipelinesv1 and piplines respectively, using this code :
credentials, _ = google.auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform'])
authed_session = google.auth.transport.requests.AuthorizedSession(
credentials)
# project_id = 'YOUR_PROJECT_ID'
# location = 'us-central1'
# composer_environment = 'YOUR_COMPOSER_ENVIRONMENT_NAME'
environment_url = (
'https://composer.googleapis.com/v1beta1/projects/{}/locations/{}'
'/environments/{}').format(project_id, location, composer_environment)
composer_response = authed_session.request('GET', environment_url)
environment_data = composer_response.json()
and the two are using the same service accounts to run, i.e. the same IAM roles. Although I have noticed the following different details :
In the old environment :
"airflowUri": "https://p5<hidden_value>-tp.appspot.com",
"privateEnvironmentConfig": { "privateClusterConfig": {} },
in the new environment:
"airflowUri": "https://da<hidden_value>-tp.appspot.com",
"privateEnvironmentConfig": {
"privateClusterConfig": {},
"webServerIpv4CidrBlock": "<hidden_value>",
"cloudSqlIpv4CidrBlock": "<hidden_value>"
}
The service account that I use to make the post request has the following roles :
Cloud Functions Service Agent
Composer Administrator
Composer User
Service Account Token Creator
Service Account User
The service account that runs my composer environment has the following roles :
BigQuery Admin
Composer Worker
Service Account Token Creator
Storage Object Admin
But I am still receiving a 403 - Forbidden in the Log Explorer when the post request is made to the airflow API.
EDIT 2020-11-16 :
I've updated to the latest make_iap_request code.
I tinkered with the IAP within the security service, but I cannot find the webserver that will accept HTTP: post requests from my cloud functions... See the image bellow, anyway I added the service account to the default and CRM IAP resources just in case, but I still get this error :
Exception: Service account does not have permission to access the IAP-protected application.
The main question is: What IAP is at stake here?? And how do I add my service account as a user of this IAP.
What am I missing?
There is a configuration parameter that causes ALL requests to the API to be denied...
In the documentation, it is mentioned that we need to override the following airflow configuration :
[api]
auth_backend = airflow.api.auth.backend.deny_all
into
[api]
auth_backend = airflow.api.auth.backend.default
This detail is really important to know, and it is not mentioned in google's documentation...
Useful links :
Triggering DAGS (workflows) with GCS
make_iap_request.py repository
The code throwing the 403 is the way it used to work. There was a breaking change in the middle of 2020. Instead of using requests to make an HTTP call for the token, you should use Google's OAuth2 library:
from google.oauth2 import id_token
from google.auth.transport.requests import Request
open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
see this example
I followed the steps in Triggering DAGs and has worked in my env, please see below my recommendations.
It is a good start that the Componser Environment is up and runnning. Through the process you will only need to upload the new DAG (trigger_response_dag.py) and get the clientID (ends with .apps.googleusercontent.com) with either a python script or from the login page the first time you open Airflow UI.
In the Cloud Functions side, I noticed you have a combination of instructions for Node.js and for Python, for example, USER_AGENT is only for Node.js. And routine make_iap_request is only for python. I hope the following points helps to resolve your problem:
Service Account (SA). The Node.js code uses the default service account ${projectId}#appspot.gserviceaccount.com whose default role is Editor, meaning that it has wide access to the GCP services, including Cloud Composer. In python I think the authentication is managed somehow by client_id since a token is retrieved with id. Please ensure that the SA has this Editor role and don't forget to assign serviceAccountTokenCreator as specified in the guide.
I used Node.js 8 runtime and I noticed the user agent you are concerned it should be 'gcf-event-trigger' as it is hard coded; USER_AGENT = 'gcf-event-trigger'. In python, it seems not necesary.
By default, in the GCS trigger, the GCS Event Type is set to Archive, you need to change it to Finalize/Create. If set to Archive, the trigger won't work when you upload objects, and the DAG won't be started.
If you think your cloud function is correctly configured and an error persists, you can find its cause in your cloud function's LOGS tab in the Console. It can give you more details.
Basically, from the guide, I only had to change the following values in Node.js:
// The project that holds your function. Replace <YOUR-PROJECT-ID>
const PROJECT_ID = '<YOUR-PROJECT-ID>';
// Navigate to your webserver's login page and get this from the URL
const CLIENT_ID = '<ALPHANUMERIC>.apps.googleusercontent.com';
// This should be part of your webserver's URL in the Env's detail page: {tenant-project-id}.appspot.com.
const WEBSERVER_ID = 'v90eaaaa11113fp-tp';
// The name of the DAG you wish to trigger. It's DAG's name in the script trigger_response_dag.py you uploaded to your Env.
const DAG_NAME = 'composer_sample_trigger_response_dag';
For Python I only changed these parameters:
client_id = '<ALPHANUMERIC>.apps.googleusercontent.com'
# This should be part of your webserver's URL:
# {tenant-project-id}.appspot.com
webserver_id = 'v90eaaaa11113fp-tp'
# Change dag_name only if you are not using the example
dag_name = 'composer_sample_trigger_response_dag'
I have faced the following problem after moving my bot to the new server. I use webhook to get updates but now the bot does not get them from telegram servers. I tried to send POST request with curl from the remove server and bot handled it in a normal way. I checked webhook with getWebhookInfo and it returned an object with non-empty url and pending_update_count equal to 74 without errors. I guess, it means that telegram servers are not able to send POST request to my host for some reason.
OS of my server is Arch Linux.
I use pyTelegramBotAP.
CONFIG = ConfigParser()
CONFIG.read(os.path.join('data', 'config.ini'))
# webhook url
URL_BASE = "https://{}:{}".format(CONFIG['server']['ip'], CONFIG.getint('server', 'port'))
URL_PATH = "/{}/".format(CONFIG['telegram bot']['token'])
BOT = telebot.TeleBot(CONFIG['telegram bot']['token'])
# server that will listen for new messages
APP = web.Application()
URL = URL_BASE + URL_PATH
BOT.set_webhook(url=URL, certificate=open(CONFIG['ssl']['certificate'], 'rb'))
# Build ssl context
CONTEXT = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
CONTEXT.load_cert_chain(CONFIG['ssl']['certificate'], CONFIG['ssl']['private key'])
# Start aiohttp server
web.run_app(
APP,
host=CONFIG['server']['listen'],
port=CONFIG['server']['port'],
ssl_context=CONTEXT,
)
Please, help!
Telegram webhook only talks to https endpoints, so I suggest to check your server against https connections.
Also, getWebhookInfo call returns a status object with the latest error infomation of your endpoint. Have a look of that error info and might find the exact problem.
Please check the Firewall on your server, It is quite possible firewall on your server is not passing message to your application.
To check firewall status run $ ufw status
Please show you URL_BASE without real IP.
What operating system is on your server?
You send test request from the CURL to URL of the Telegram or of your server?
Can you getting response from your server if you run simple app?
Example:
from aiohttp import web
async def hello(request):
return web.Response(text='Hello world!')
app = web.Application()
app.add_routes([web.get('/', hello)])
web.run_app(app, host='localhost', port=3003)
Check response:
$ curl localhost:3003
Hello world!
Please provide more detailed information on how you troubleshoot.
I am trying to set a cookie with Flask after login and redirect on the front end in Javascript.
#app.route("/login")
#auth.login_required
def get_auth_token():
token = g.user.generate_auth_token()
request = make_response()
token = str(token.decode("ascii"))
request.set_cookie("token", value = token)
return request, 200
No matter if I have the redirect in or not, the cookie never sets. I've tried commenting out my redirect on the front end, I've tried setting my cookie with secure = false but none of that seems to work. What am I missing? If needed, I can provide the generate_suth_token function, but I know that is working properly. I am serving on localhost:5000 and using Flask 0.12.2, and received no cookie warnings in the server log.
If Flask service and client service are being hosted on different domains (e. g Flask uses 127.0.0.1:8080 and a client uses 127.0.0.1:3000) in this case, cookies should be set with domain parameter otherwise they will not be available.
resp.set_cookie('cookie_key', value="cookie_value", domain='127.0.0.1')
Find more info about domain parameter here
I'm testing the Instagram Real-time API with Python and Flask and I get everytime this response from the Instagram server:
{
"meta":{
"error_type":"APISubscriptionError",
"code":400,
"error_message":"Unable to reach callback URL \"http:\/\/my_callback_url:8543\/instagram\"."
}
}
The request:
curl -F 'client_id=my_client_id...' \
-F 'client_secret=my_client_secret...' \
-F 'object=tag' \
-F 'aspect=media' \
-F 'object_id=fox' \
-F 'callback_url=http://my_callback_url:8543/instagram' \
https://api.instagram.com/v1/subscriptions/
And this is the code of the Flask server:
from flask import Flask
from flask import request
from instagram import subscriptions
app = Flask(__name__)
CLIENT_ID = "my_client_id..."
CLIENT_SECRET = "my_client_secret..."
def process_tag_update(update):
print 'Received a push: ', update
reactor = subscriptions.SubscriptionsReactor()
reactor.register_callback(subscriptions.SubscriptionType.TAG, process_tag_update)
#app.route('/instagram', methods=['GET'])
def handshake():
# GET method is used when validating the endpoint as per the Pubsubhubub protocol
mode = request.values.get('hub.mode')
challenge = request.values.get('hub.challenge')
verify_token = request.values.get('hub.verify_token')
if challenge:
return challenge
return 'It is not a valid challenge'
#app.route('/instagram', methods=['POST'])
def callback():
# POST event is used to for the events notifications
x_hub_signature = request.headers.get('X-Hub-Signature')
raw_response = request.data
try:
reactor.process(CLIENT_SECRET, raw_response, x_hub_signature)
except subscriptions.SubscriptionVerifyError:
print 'Signature mismatch'
return 'done'
def server():
""" Main server, will allow us to make it wsgi'able """
app.run(host='0.0.0.0', port=8543, debug=True)
if __name__ == '__main__':
server()
The machine have a public IP and the port is open for everyone. I can reach the url from others networks.
Why can't Instagram reach my url? Is there a black list or something like that?
Update 1
I have tested the same code with some frameworks and WSGI servers (Django, Flask, Node.js, Gunicorn, Apache) and different responses in the GET/POST endpoint and I always get the same 400 error message.
Also I have checked the packages received in my network interface with Wireshark and I get the expected results with calls from any network. But I don't get any package when I do the subscription request.
So... Is this a bug? Could be my IP in any blacklist for some reason?
I had exactly the same. It worked when I accidentally restarted the router, getting a different IP. It seems that it could be an IP issue indeed and the Unable to reach callback URL... is not really helpful in this case.
I agree, there are plenty of AWS servers answering to that API and some are not working. Ping api.instagram.com and you'll see you get multiple and different IP for that domain name. There is a DNS round robin and you are not reaching the same server every time.
I've found one server (IP : 52.6.133.72) which seems to be working for subscription and have configured my server to use that one (by editing the /etc/hosts file). Not a reliable solution ... but it's working.
Github offers to send Post-receive hooks to an URL of your choice when there's activity on your repo.
I want to write a small Python command-line/background (i.e. no GUI or webapp) application running on my computer (later on a NAS), which continually listens for those incoming POST requests, and once a POST is received from Github, it processes the JSON information contained within. Processing the json as soon as I have it is no problem.
The POST can come from a small number of IPs given by github; I plan/hope to specify a port on my computer where it should get sent.
The problem is, I don't know enough about web technologies to deal with the vast number of options you find when searching.. do I use Django, Requests, sockets,Flask, microframeworks...? I don't know what most of the terms involved mean, and most sound like they offer too much/are too big to solve my problem - I'm simply overwhelmed and don't know where to start.
Most tutorials about POST/GET I could find seem to be concerned with either sending or directly requesting data from a website, but not with continually listening for it.
I feel the problem is not really a difficult one, and will boil down to a couple of lines, once I know where to go/how to do it. Can anybody offer pointers/tutorials/examples/sample code?
First thing is, web is request-response based. So something will request your link, and you will respond accordingly. Your server application will be continuously listening on a port; that you don't have to worry about.
Here is the similar version in Flask (my micro framework of choice):
from flask import Flask, request
import json
app = Flask(__name__)
#app.route('/',methods=['POST'])
def foo():
data = json.loads(request.data)
print "New commit by: {}".format(data['commits'][0]['author']['name'])
return "OK"
if __name__ == '__main__':
app.run()
Here is a sample run, using the example from github:
Running the server (the above code is saved in sample.py):
burhan#lenux:~$ python sample.py
* Running on http://127.0.0.1:5000/
Here is a request to the server, basically what github will do:
burhan#lenux:~$ http POST http://127.0.0.1:5000 < sample.json
HTTP/1.0 200 OK
Content-Length: 2
Content-Type: text/html; charset=utf-8
Date: Sun, 27 Jan 2013 19:07:56 GMT
Server: Werkzeug/0.8.3 Python/2.7.3
OK # <-- this is the response the client gets
Here is the output at the server:
New commit by: Chris Wanstrath
127.0.0.1 - - [27/Jan/2013 22:07:56] "POST / HTTP/1.1" 200 -
Here's a basic web.py example for receiving data via POST and doing something with it (in this case, just printing it to stdout):
import web
urls = ('/.*', 'hooks')
app = web.application(urls, globals())
class hooks:
def POST(self):
data = web.data()
print
print 'DATA RECEIVED:'
print data
print
return 'OK'
if __name__ == '__main__':
app.run()
I POSTed some data to it using hurl.it (after forwarding 8080 on my router), and saw the following output:
$ python hooks.py
http://0.0.0.0:8080/
DATA RECEIVED:
test=thisisatest&test2=25
50.19.170.198:33407 - - [27/Jan/2013 10:18:37] "HTTP/1.1 POST /hooks" - 200 OK
You should be able to swap out the print statements for your JSON processing.
To specify the port number, call the script with an extra argument:
$ python hooks.py 1234
I would use:
https://github.com/carlos-jenkins/python-github-webhooks
You can configure a web server to use it, or if you just need a process running there without a web server you can launch the integrated server:
python webhooks.py
This will allow you to do everything you said you need. It, nevertheless, requires a bit of setup in your repository and in your hooks.
Late to the party and shameless autopromotion, sorry.
If you are using Flask, here's a very minimal code to listen for webhooks:
from flask import Flask, request, Response
app = Flask(__name__)
#app.route('/webhook', methods=['POST'])
def respond():
print(request.json) # Handle webhook request here
return Response(status=200)
And the same example using Django:
from django.http import HttpResponse
from django.views.decorators.http import require_POST
#require_POST
def example(request):
print(request.json) # Handle webhook request here
return HttpResponse('Hello, world. This is the webhook response.')
If you need more information, here's a great tutorial on how to listen for webhooks with Python.
If you're looking to watch for changes in any repo...
1. If you own the repo that you want to watch
In your repo page, Go to settings
click webhooks, new webhook (top right)
give it your ip/endpoint and setup everything to your liking
use any server to get notified
2. Not your Repo
take the url you want i.e https://github.com/fire17/gd-xo/
add /commits/master.atom to the end such as:
https://github.com/fire17/gd-xo/commits/master.atom
Use any library you want to get that page's content, like:
filter out the keys you want, for example the element
response = requests.get("https://github.com/fire17/gd-xo/commits/master.atom").text
response.split("<updated>")[1].split("</updated>")[0]
'2021-08-06T19:01:53Z'
make a loop that checks this every so often and if this string has changed, then you can initiate a clone/pull request or do whatever you like