Develop Reference

extend dialogflow webhook deadline time for gpt api call

I am trying to use a script I found on the internet to extend the maximum time for a webhook request through Google Dialogflow (max 5 seconds to timeout). I need to extend the time because I make an API call to openai and it sometimes takes longer than 5 seconds. My idea was to start the 2 functions in parallel. The broadbridge_webhook_results() function is there to extend the time by triggering a followupEventInput at Dialogflow after 3,5 seconds, so a new call comes through Dialogflow and the 5 seconds start from new. This goes apparently up to 2 times. In the meantime the API call should be made towards openai. As soon as the API call was successful, the answer should be sent back to Dialogflow. Unfortunately, I am currently not getting anywhere and I think that the thread functionality was set up or understood incorrectly by me.
The following code I have so far:
import os
import openai
import time
import backoff
from datetime import datetime, timedelta
from flask import Flask, request, render_template
from threading import Thread
import asyncio
app = Flask(__name__)
conversation_History = ""
user_Input = ""
reply=''
answer = ""
#app.route('/')
def Default():
return render_template('index.html')
#backoff.on_exception(backoff.expo, openai.error.RateLimitError)
def ask(question):
global conversation_History
global answer
global reply
openai.api_key = os.getenv("gtp_Secret_Key")
#start_sequence = "\nAI:"
#restart_sequence = "\nHuman: "
response = openai.Completion.create(
model="text-davinci-003",
prompt="I am a chatbot from OpenAI. I'm happy to answer your questions.\nHuman:" + conversation_History + " "+ question +"\nAI: ",
temperature=0.9,
max_tokens=500,
top_p=1,
frequency_penalty=0,
presence_penalty=0.6,
stop=[" Human:", " AI:"]
)
conversation_History = conversation_History + question + "\nAI" + answer + "\nHuman:"
answer = response.choices[0].text
def broadbridge_webhook_results():
global answer
now = datetime.now()
current_time = now.strftime("%H:%M:%S")
print("Current Time =", current_time)
extended_time = now + timedelta(seconds=3)
print("extended Time =", extended_time.time())
req = request.get_json(force=True)
action = req.get('queryResult').get('action')
reply=''
if action=='input.unknown' or action=='input.welcome':
time.sleep(3.5)
if now<=extended_time and not len(answer) == 0:
reply={
"fulfillmentText": answer,
"source": "webhookdata"
}
reply={
"followupEventInput": {
"name": "extent_webhook_deadline",
"languageCode": "en-US"
}
}
if action=='followupevent':
print("enter into first followup event")
time.sleep(3.5)
if now<=extended_time and not len(answer) == 0:
reply={
"fulfillmentText": answer,
"source": "webhookdata"
}
reply={
"followupEventInput": {
"name": "extent_webhook_deadline_2",
"languageCode": "en-US"
}
}
if action=='followupevent_2':
print("enter into second followup event")
time.sleep(3.5)
reply={
"fulfillmentText": answer,
"source": "webhookdata"
}
print("Final time of execution:=>", now.strftime("%H:%M:%S"))
#app.route('/webhook', methods=['GET', 'POST'])
def webhook():
global answer
global reply
answer=""
req = request.get_json(silent=True, force=True)
user_Input = req.get('queryResult').get('queryText')
Thread(target=broadbridge_webhook_results()).start()
Thread(target=ask(user_Input)).start()
return reply
#conversation_History = conversation_History + user_Input + "\nAI" + answer + "\nHuman:"
#if now<=extended_time and not len(answer) == 0:
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080)

Dialogflow supports increasing the webhook execution time limit up to 30 seconds by increasing the webhook execution time limit. This is done through the "extend_webhook_deadline" setting in the Dialogflow agent configuration file.
To enable this setting you will need to do the following:
Access the Dialogflow Console
Select the agent for which you want to enable the setting
Click "Settings" on the left sidebar
Click "Export and Import"
Select "Export as ZIP"
Locate the "agent.json" file inside the downloaded ZIP file
Open the "agent.json" file with a text editor
Locate a "greeting" section
Add the following line "extend_webhook_deadline": true
Save the changes and import the "agent.json" file back into Dialogflow
Bear in mind that by enabling this setting, Dialogflow will charge an additional fee to increase the duration of the webhook.

eBay Marketplace Account Deletion/Closure Notifications

I am trying to start an eBay API in Python and I can't find a single answer as to how to get an API key with eBay's new requirements of "Account Deletion/Closure Notifications." Here's the link: https://developer.ebay.com/marketplace-account-deletion
Specifically, I am told that "Your Keyset is currently disabled" because I have not completed whatever process is needed for this marketplace account deletion/closure notification.
The problems?
I have no idea if I need this.
I have no idea how to actually do this.
Re: 1. It looks like this is for anyone who stores user data. I don’t think that’s me intentionally because I really just want to get sold data and current listings, but is it actually me?
Re: 2. I don’t understand how to validate it and send back the proper responses. I’ve gotten quite good at python but I’m lost here.
eBay forums are completely useless and I see no one with an answer to this. Any help is greatly appreciated.

Re: 1. Same. Here's my interpretation: In order to use their APIs, you need to provide (and configure) your own API, so they can communicate with you —programatically— and tell you what users have asked to have their accounts/data deleted.
Re: 2. To handle their GET and POST requests, I guess you'll need to configure a website's URL as an API endpoint. In Django, I might use something like this (untested) code:
import hashlib
import json
from django.http import (
HttpResponse,
JsonResponse,
HttpResponseBadRequest
)
def your_api_endpoint(request):
"""
API Endpoint to handle the verification's challenge code and
receive eBay's Marketplace Account Deletion/Closure Notifications.
"""
# STEP 1: Handle verification's challenge code
challengeCode = request.GET.get('challenge_code')
if challengeCode is not None:
# Token needs to be 32-80 characters long
verificationToken = "your-token-012345678901234567890123456789"
# URL needs to use HTTPS protocol
endpoint_url = "https://your-domain.com/your-endpoint"
# Hash elements need to be ordered as follows
m = hashlib.sha256((challengeCode+verificationToken+endpoint_url).encode('utf-8'))
# JSON field needs to be called challengeResponse
return JsonResponse({"challengeResponse": m.hexdigest()}, status=200)
# STEP 2: Handle account deletion/closure notification
elif request.method == 'POST':
notification_details = json.loads(request.body)
# Verify notification is actually from eBay
# ...
# Delete/close account
# ...
# Acknowledge notification reception
return HttpResponse(status=200)
else:
return HttpResponseBadRequest()
If you find the answer to question number one, please do let me know.

Re: 1. You need to comply with eBay's Marketplace Account Deletion/Closure Notification workflow if you are storing user data into your own database. For example, using eBay's Buy APIs, you may get access to what users are selling on eBay (for ex. an eBay feed of products). If those eBay sellers decide they want to remove all of their personal data from eBay's database, eBay is requesting you remove their data from your database as well. If you are NOT storing any eBay user data into your database, you do not need to comply. Here is where you can find more info: https://partnerhelp.ebay.com/helpcenter/s/article/Complying-with-the-eBay-Marketplace-Account-Deletion-Closure-Notification-workflow?language=en_US
Re: 2. To be honest I've spent days trying to figure this out in Python (Django), but I have a solution now and am happy to share it with whoever else comes across this issue. Here's my solution:
import os
import json
import base64
import hashlib
import requests
import logging
from OpenSSL import crypto
from rest_framework import status
from rest_framework.views import APIView
from django.http import JsonResponse
logger = logging.getLogger(__name__)
class EbayMarketplaceAccountDeletion(APIView):
"""
This is required as per eBay Marketplace Account Deletion Requirements.
See documentation here: https://developer.ebay.com/marketplace-account-deletion
"""
# Ebay Config Values
CHALLENGE_CODE = 'challenge_code'
VERIFICATION_TOKEN = os.environ.get('VERIFICATION_TOKEN')
# ^ NOTE: You can make this value up so long as it is between 32-80 characters.
ENDPOINT = 'https://example.com/ebay_marketplace_account_deletion'
# ^ NOTE: Replace this with your own endpoint
X_EBAY_SIGNATURE = 'X-Ebay-Signature'
EBAY_BASE64_AUTHORIZATION_TOKEN = os.environ.get('EBAY_BASE64_AUTHORIZATION_TOKEN')
# ^ NOTE: Here's how you can get your EBAY_BASE64_AUTHORIZATION_TOKEN:
# import base64
# base64.b64encode(b'{CLIENT_ID}:{CLIENT_SECRET}')
def __init__(self):
super(EbayMarketplaceAccountDeletion, self).__init__()
def get(self, request):
"""
Get challenge code and return challengeResponse: challengeCode + verificationToken + endpoint
:return: Response
"""
challenge_code = request.GET.get(self.CHALLENGE_CODE)
challenge_response = hashlib.sha256(challenge_code.encode('utf-8') +
self.VERIFICATION_TOKEN.encode('utf-8') +
self.ENDPOINT.encode('utf-8'))
response_parameters = {
"challengeResponse": challenge_response.hexdigest()
}
return JsonResponse(response_parameters, status=status.HTTP_200_OK)
def post(self, request):
"""
Return 200 status code and remove from db.
See how to validate the notification here:
https://developer.ebay.com/api-docs/commerce/notification/overview.html#use
"""
# Verify notification is actually from eBay #
# 1. Use a Base64 function to decode the X-EBAY-SIGNATURE header and retrieve the public key ID and signature
x_ebay_signature = request.headers[self.X_EBAY_SIGNATURE]
x_ebay_signature_decoded = json.loads(base64.b64decode(x_ebay_signature).decode('utf-8'))
kid = x_ebay_signature_decoded['kid']
signature = x_ebay_signature_decoded['signature']
# 2. Call the getPublicKey Notification API method, passing in the public key ID ("kid") retrieved from the
# decoded signature header. Documentation on getPublicKey:
# https://developer.ebay.com/api-docs/commerce/notification/resources/public_key/methods/getPublicKey
public_key = None
try:
ebay_verification_url = f'https://api.ebay.com/commerce/notification/v1/public_key/{kid}'
oauth_access_token = self.get_oauth_token()
headers = {
'Authorization': f'Bearer {oauth_access_token}'
}
public_key_request = requests.get(url=ebay_verification_url, headers=headers, data={})
if public_key_request.status_code == 200:
public_key_response = public_key_request.json()
public_key = public_key_response['key']
except Exception as e:
message_title = "Ebay Marketplace Account Deletion: Error calling getPublicKey Notfication API."
logger.error(f"{message_title} Error: {e}")
return JsonResponse({}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
# 3. Initialize the cryptographic library to perform the verification with the public key that is returned from
# the getPublicKey method. If the signature verification fails, an HTTP status of 412 Precondition Failed is returned.
pkey = crypto.load_publickey(crypto.FILETYPE_PEM, self.get_public_key_into_proper_format(public_key))
certification = crypto.X509()
certification.set_pubkey(pkey)
notification_payload = request.body
signature_decoded = base64.b64decode(signature)
try:
crypto.verify(certification, signature_decoded, notification_payload, 'sha1')
except crypto.Error as e:
message_title = f"Ebay Marketplace Account Deletion: Signature Invalid. " \
f"The signature is invalid or there is a problem verifying the signature. "
logger.warning(f"{message_title} Error: {e}")
return JsonResponse({}, status=status.HTTP_412_PRECONDITION_FAILED)
except Exception as e:
message_title = f"Ebay Marketplace Account Deletion: Error performing cryptographic validation."
logger.error(f"{message_title} Error: {e}")
return JsonResponse({}, status=status.HTTP_412_PRECONDITION_FAILED)
# Take appropriate action to delete the user data. Deletion should be done in a manner such that even the
# highest system privilege cannot reverse the deletion #
# TODO: Replace with your own data removal here
# Acknowledge notification reception
return JsonResponse({}, status=status.HTTP_200_OK)
def get_oauth_token(self):
"""
Returns the OAuth Token from eBay which can be used for making other API requests such as getPublicKey
"""
url = 'https://api.ebay.com/identity/v1/oauth2/token'
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': f"Basic {self.EBAY_BASE64_AUTHORIZATION_TOKEN}"
}
payload = 'grant_type=client_credentials&scope=https%3A%2F%2Fapi.ebay.com%2Foauth%2Fapi_scope'
request = requests.post(url=url, headers=headers, data=payload)
data = request.json()
return data['access_token']
#staticmethod
def get_public_key_into_proper_format(public_key):
"""
Public key needs to have \n in places to be properly assessed by crypto library.
"""
return public_key[:26] + '\n' + public_key[26:-24] + '\n' + public_key[-24:]

This is how I am dealing with the ebay notification requirement using Python3 cgi. Because bytes are sent, cannot use cgi.FieldStorage()
import os
import sys
import hashlib
import json
from datetime import datetime
from html import escape
import cgi
import cgitb
import io
include_path = '/var/domain_name/www'
sys.path.insert(0, include_path)
cgitb.enable(display=0, logdir=f"""{include_path}/tmp_errors""") # include_path is OUTDIR
dt_now = datetime.now()
current_dt_now = dt_now.strftime("%Y-%m-%d_%H-%M-%S")
def enc_print(string='', encoding='utf8'):
sys.stdout.buffer.write(string.encode(encoding) + b'\n')
html = ''
challengeCode = ''
# GET
myQuery = os.environ.get('QUERY_STRING')
if myQuery.find('=') != -1:
pos = myQuery.find('=')
var_name = myQuery[:pos]
var_val = myQuery[pos+1:]
challengeCode = var_val
# POST
if os.environ.get('CONTENT_LENGTH') != None:
totalBytes=int(os.environ.get('CONTENT_LENGTH'))
reqbytes=io.open(sys.stdin.fileno(),"rb").read(totalBytes)
if challengeCode != '' :
"""
API Endpoint to handle the verification's challenge code and
receive eBay's Marketplace Account Deletion/Closure Notifications.
"""
# STEP 1: Handle verification's challenge code
# Token needs to be 32-80 characters long
verificationToken = "0123456789012345678901234567890123456789" #sample token
# URL needs to use HTTPS protocol
endpoint = "https://domain_name.com/ebay/notification.py" # sample endpoint
# Hash elements need to be ordered as follows
m = hashlib.sha256( (challengeCode+verificationToken+endpoint).encode('utf-8') )
# JSON field needs to be called challengeResponse
enc_print("Content-Type: application/json")
enc_print("Status: 200 OK")
enc_print()
enc_print('{"challengeResponse":"' + m.hexdigest() + '"}')
exit()
else :
#html += 'var length:' + str(totalBytes) + '\n'
html += reqbytes.decode('utf-8') + '\n'
# STEP 2: Handle account deletion/closure notification
# Verify notification is actually from eBay
# ...
# Delete/close account
# ...
# Acknowledge notification reception
with open( f"""./notifications/{current_dt_now}_user_notification.txt""", 'w') as f:
f.write(html)
enc_print("Content-Type: application/json")
enc_print("Status: 200 OK")
enc_print()
exit()

I've been trying #José Matías Arévalo code. It works except "STEP 2" branch - Django returns 403 error. This is because of by default Django uses CSRF middleware (Cross Site Request Forgery protection). To avoid 403 error we need to marks a view as being exempt from the protection as described here https://docs.djangoproject.com/en/dev/ref/csrf/#utilities so add couple strings in code:
from django.views.decorators.csrf import csrf_exempt
#csrf_exempt
def your_api_endpoint(request):
And in my case I use url "https://your-domain.com/your-endpoint/" with slash symbol "/" at the end of url. Without this slash eBay doesn't confirm subscription.

I am using Flask and this is the code I have used:
from flask import Flask, request
import hashlib
# Create a random verification token, it needs to be 32-80 characters long
verification_token = 'a94cbd68e463cb9780e2008b1f61986110a5fd0ff8b99c9cba15f1f802ad65f9'
endpoint_url = 'https://dev.example.com'
app = Flask(__name__)
# There will be errors if you just use '/' as the route as it will redirect eBays request
# eBay will send a request to https://dev.example.com?challenge_code=123
# The request will get redirected by Flask to https://dev.example.com/?challenge_code=123 which eBay will not accept
endpoint = endpoint_url + '/test'
# The Content-Type header will be added automatically by Flask as 'application/json'
#app.route('/test')
def test():
code = request.args.get('challenge_code')
print('Requests argument:', code)
code = code + token + endpoint
code = code.encode('utf-8')
code = hashlib.sha256(code)
code = code.hexdigest()
print('Hexdigest:', code)
final = {"challengeResponse": code}
return final
## To run locally first use this:
# app.run(port=29)

Failed messages on Twilio

I built a whatapp chatbot using flask and Twilio, but seem to getting failed messages on when I check my console. The status appears as failed with an unknown error. When I check my debugger I see the error message
HTTP retrieval failure
I started getting these errors after making a few changes to my flask app. I tried undoing all the changes but now instead of getting received as the status, the status now appears as failed. I tried restarted my ngrok and changing the url on my Twilio but this hasn't changed anything either.
Here is my code:
from flask import Flask, request
import requests
from twilio.twiml.messaging_response import MessagingResponse
import random
app = Flask(__name__)
#app.route('/bot', methods=['POST'])
def bot():
incoming_msg = request.values.get('Body', '').lower()
resp = MessagingResponse()
msg = resp.message()
responded = False
if incoming_msg == 'help':
#return options available
output = 'This is a chatbot designed to send statistics, linear algebra and general programming problem questions. Please type in either "Python", "Statistics" or Linear Algebra" to get a random problem.'
msg.body(output)
responded = True
if 'python' in incoming_msg:
with open("C://Users//User//Downloads//python_test.txt", "r") as f:
lines = f.readlines()
code = random.choice(lines)
msg.body(code)
responded = True
if 'quote' in incoming_msg:
# return a quote
r = requests.get('https://api.quotable.io/random')
if r.status_code == 200:
data = r.json()
quote = f'{data["content"]} ({data["author"]})'
else:
quote = 'I could not retrieve a quote at this time, sorry.'
msg.body(quote)
responded = True
if 'thegradientboost' in incoming_msg:
message = 'Test \
'. Additional text' \
'More text,' \
'end of text.' \
'a bit more text' \
'conclusion.'
msg.body(message)
responded = True
if not responded:
msg.body('Word not included.')
return str(resp)
if __name__ == '__main__':
app.run(debug=True)

Slack Interactive Messages: POST request payload has an unexpected format

I'm getting a POST request inside a Flask app from Slack. The request is sent when a user presses on an interactive message button. According to Slack docs I must extract the body of the request to verify the signature.
My computed signature doesn't match the one sent by Slack, though.
In fact, the body of the request comes as some encoded string. The string is actually an encoded dictionary instead of a query str parameters, as expected.
Here's the beginning of my view:
#app.route('/register', methods=['POST'])
def register_visit():
data = request.get_data()
signature = request.headers.get('X-Slack-Signature', None)
timestamp = request.headers.get('X-Slack-Request-Timestamp', None)
signing_secret = b'aaaaaaaaaaaaaaaa'
# old message, ignore
if round(actual_time.time() - float(timestamp)) > 60 * 5:
return
concatenated = ("v0:%s:%s" % (timestamp, data)).encode('utf-8')
computed_signature = 'v0=' + hmac.new(signing_secret, msg=concatenated, digestmod=hashlib.sha256).hexdigest()
if hmac.compare_digest(computed_signature, signature):
...
I've tried to format the received data to make it look like:
token=fdjkgjl&user_id=1234... but I am not aware of all of the necessary parameters that have to be present in the data.
Any ideas are highly appreciated.
The body of the message is following - after being URL decoded (note I've modified possibly sensitive data):
b'payload={"type":"interactive_message","actions":
[{"name":"yes_button","type":"button","value":"236"}],"callback_id":"visit_button","team":{"id":"fffff","domain":"ffff"},"channel":{"id":"ffff","name":"directmessage"},"user":{"id":"ffffff","name":"fffft"},"action_ts":"1540403943.419120","message_ts":"1541403949.000100","attachment_id":"1","token":"8LpjBuv13J7xAjhl2lEajoBU","is_app_unfurl":false,"original_message":{"text":"Test","bot_id":"DDDDDDDDD","attachments":[{"callback_id":"visit_button","text":"Register","id":1,"color":"3AA3E3","actions":[{"id":"1","name":"yes_button","text":"Yes","type":"button","value":"236","style":""}],"fallback":"Register"}],"type":"message","subtype":"bot_message","ts":"1540413949.000100"},"response_url":"https://hooks.slack.com/actions/ffffff/ffffff/tXJjx1XInaUhrikj6oEzK08e","trigger_id":"464662548327.425084163429.dda35a299eedb940ab98dbb9386b56f0"}'

The reason you are getting the "garbled" data is that you are using request.get_data(). That method will return the raw data of a request, but not do any decoding for you.
Much more convenient is to use request.form.get('payload'), which will directly give you the JSON string of the request object. You can then convert that into a dict object with json.loads() to process it further in your app.
Note that the format you received is the correct format for interactive messages. You will not get a query string (e.g. "token=abc;user_id?def...") as you suggested (like for slash command requests). Interactive message request will always contain the request as JSON string in a payload form property. See here for reference.
Here is a simple working example, which will reply a greeting to the user that pressed the button. It will work directly with Slack, but I recommend using Postman to test it.
#app.py
from flask import Flask, request #import main Flask class and request object
import json
app = Flask(__name__) #create the Flask app
#app.route('/register', methods=['POST'])
def register_visit():
slack_req = json.loads(request.form.get('payload'))
response = '{"text": "Hi, <#' + slack_req["user"]["id"] + '>"}'
return response, 200, {'content-type': 'application/json'}
if __name__ == '__main__':
app.run(debug=True, port=5000) #run app in debug mode on port 5000

OK, the issue wasn't related to how Slack sends me the message. It was about misunderstanding which data comes as bytes and which data is unicode. The culprit was string formatting in my case - the line concatenated = ("v0:%s:%s" % (timestamp, data)).encode('utf-8') should have been concatenated = (b"v0:%b:%b" % (timestamp.encode("utf-8"), data)). Data is already bytes, timestamp meanwhile is unicode.
Cannot believe I've banged my head on this for hours -_-
#app.route('/register', methods=['POST'])
def register_visit():
data = request.get_data()
signature = request.headers.get('X-Slack-Signature', None)
timestamp = request.headers.get('X-Slack-Request-Timestamp', None)
signing_secret = b'aaaaaaaaaaaaaaaa'
# old message, ignore
if round(actual_time.time() - float(timestamp)) > 60 * 5:
return
concatenated = (b"v0:%b:%b" % (timestamp.encode("utf-8"), data))
computed_signature = 'v0=' + hmac.new(signing_secret, msg=concatenated,
digestmod=hashlib.sha256).hexdigest()
if hmac.compare_digest(computed_signature, signature):
...

This worked for me
from urllib import parse
parsed_text = parse.unquote('your bytes text here')

Set a cookie and retrieve it with Python and WSGI

a lot of questions exists that are similar to this, but none of them helped me out. Basically I'm using WSGI start_response() method link. I tried to set a dummy header in the response with the tuple [('Set-Cookie', 'token=THE_TOKEN')] and add it to start response like this:
status = '200 OK'
response = 'Success'
start_response(status,[('Set-Cookie', "DMR_TOKEN=DMR_TOKEN")])
return response
I'm not pretty sure that is working correctly, but it's here setting cookies. Now, let's suppose the header is correct and in following requests I want to authenticate a token. What would be the correct way to catch that cookie/header setted in the past ?
I've been reading and find I need something like this:
(environ.get("HTTP_COOKIE",""))
but that has been yielding empty string all the time, so I'm just assuming the header/cookie is not correctly set.
Thanks guys

I think you need to set the path explicitly to get useful behavior out of cookies, try something like:...
from Cookie import SimpleCookie
def my_app(environ, start_response):
session_cookie = SimpleCookie()
session_cookie['session'] = "somedata"
session_cookie['session']["Path"] = '/'
headers = []
headers.extend(("set-cookie", morsel.OutputString())
for morsel
in session_cookie.values())
start_response("200 OK", headers)

Here's a simple solution I came up with for setting cookies in WSGI that doesn't require an external library:
def set_cookie_header(name, value, days=365):
dt = datetime.datetime.now() + datetime.timedelta(days=days)
fdt = dt.strftime('%a, %d %b %Y %H:%M:%S GMT')
secs = days * 86400
return ('Set-Cookie', '{}={}; Expires={}; Max-Age={}; Path=/'.format(name, value, fdt, secs))
def handler(env, start_response):
content_type = 'text/html'
headers = [('Content-Type', content_type), set_cookie_header('name', 'value')]
start_response('200 OK', headers)
...