This is my first project using API's so apologies if this is a silly question to ask!
I've written some code that has an input of a playlist url from Spotify. It then creates a new playlist on YouTube and adds the songs from the Spotify playlist into the YouTube one.
I had just got it to work perfectly. Then all I did was input a new Spotify url and it started giving me this error which I can't get rid of.
HttpError: <HttpError 500 when requesting https://www.googleapis.com/youtube/v3/playlists?part=snippet%2Cstatus&alt=json returned "Internal error encountered.">
Could anyone tell me what the problem is? Thanks so much!
**EDIT: I went to the location of the error and this is the bit of code that is giving the error (I didn't write this bit, this is from HTTP requests)
def execute(self, http=None, num_retries=0):
"""Execute the request.
Args:
http: httplib2.Http, an http object to be used in place of the
one the HttpRequest request object was constructed with.
num_retries: Integer, number of times to retry with randomized
exponential backoff. If all retries fail, the raised HttpError
represents the last request. If zero (default), we attempt the
request only once.
Returns:
A deserialized object model of the response body as determined
by the postproc.
Raises:
googleapiclient.errors.HttpError if the response was not a 2xx.
httplib2.HttpLib2Error if a transport error has occurred.
"""
if http is None:
http = self.http
if self.resumable:
body = None
while body is None:
_, body = self.next_chunk(http=http, num_retries=num_retries)
return body
# Non-resumable case.
if "content-length" not in self.headers:
self.headers["content-length"] = str(self.body_size)
# If the request URI is too long then turn it into a POST request.
# Assume that a GET request never contains a request body.
if len(self.uri) > MAX_URI_LENGTH and self.method == "GET":
self.method = "POST"
self.headers["x-http-method-override"] = "GET"
self.headers["content-type"] = "application/x-www-form-urlencoded"
parsed = urlparse(self.uri)
self.uri = urlunparse(
(parsed.scheme, parsed.netloc, parsed.path, parsed.params, None, None)
)
self.body = parsed.query
self.headers["content-length"] = str(len(self.body))
# Handle retries for server-side errors.
resp, content = _retry_request(
http,
num_retries,
"request",
self._sleep,
self._rand,
str(self.uri),
method=str(self.method),
body=self.body,
headers=self.headers,
)
for callback in self.response_callbacks:
callback(resp)
if resp.status >= 300:
raise HttpError(resp, content, uri=self.uri)
return self.postproc(resp, content)
Related
I have used the following guide - https://developer.ebay.com/marketplace-account-deletion
So I've created a flask based website on python and uploaded in Heroku,
https://ebay-deletion.herokuapp.com/
If you create a query parameter, for the challenge code like so:
https://ebay-deletion.herokuapp.com/?challenge_code=123
It returns a challenge response and doc type (JSON).
I don't understand but why but I get this error below:
Notification delivery failed with HTTP status code 405 from https://ebay-deletion.herokuapp.com. Please ensure that the marketplace account deletion notification endpoint is ready to receive notifications.
Any thoughts on how to solve this?
Ebay's help page is just so terrible. Their python code examples for hashing in the top link have semi-colons after each line! It's close to completely useless.
Here is the Python Flask code:
#app.route('/')
def index():
args = request.args
args_dict = args.to_dict()
try:
resp_hash = hashlib.sha256(
args_dict['challenge_code'].encode() + verification_token.encode() + endpoint.encode())
resp = {'challengeResponse': resp_hash.hexdigest()}
return jsonify(resp), 200, {'content-type': 'application/json'}
except KeyError:
err = {'status_code': 400, 'message': 'no challenge response in params'}
return jsonify(err), 400, {'content-type': 'application/json'}
Currently, I have a working API that uses Connexion and receives an OpenAPI specification:
connexion_app.add_api(
"openapi.yaml",
options={"swagger_ui": False},
validate_responses=True,
strict_validation=True, # Changing this also didn't help
)
A response gets validated in the following order:
Check if API-Key is valid
Validate if the request body contains all necessary parameters
Validate message-signature
Handle request and send response
The verification of the API-Key is done via the OpenAPI spec:
securitySchemes:
apiKeyAuth:
type: apiKey
in: header
name: API-Key
x-apikeyInfoFunc: server.security.check_api_key
security:
- apiKeyAuth: []
The validation is also done via the OpenAPI spec.
The signature gets verified in the endpoint:
if not verify_signature(kwargs):
abort(401, "Signature could not be verified")
Where verify_signature is basically this:
def verify_signature(request) -> bool:
"""Calculate the signature using the header and data."""
signature = re.findall(r'"([A-Za-z0-9+/=]+)"', connexion.request.headers.get("Message-Signature", ""))
created = re.findall(r"created=(\d+)", connexion.request.headers.get("Message-Signature", ""))
if len(signature) == 0:
abort(401, "No valid Signature found.")
if len(created) == 0:
abort(401, "No valid created timestamp found.")
signature = signature[0]
created = int(created[0])
method, path, host, api_key, content_type = _get_attributes_from_request()
message = create_signature_message(request["body"], created, method, path, host, api_key, content_type)
recreated_signature = _encode_message(message)
return recreated_signature == str(signature)
For security purposes I would like to swap 2. and 3.:
Check if API-Key is valid
Validate message-signature
Validate if the request body contains all necessary parameters
Handle request and send response
The problem is that Connexion validates the body before I get to my endpoint in which I execute my Python code such as verify_signature.
I tried adding the following to my OpenAPI.yaml:
signatureAuth:
type: http
scheme: basic
x-basicInfoFunc: server.security.verify_signature
security:
- apiKeyAuth: []
signatureAuth: []
But I think this is the wrong approach since I think this is only used as a simple verification method and I get the following error message:
No authorization token provided.
Now to my question:
Is there a way to execute a function which receives the whole request that gets executed before Connexion validates the body?
Yes you can use the Connexion before_request annotation so it runs a function on a new request before validating the body. Here's an example that logs the headers and content:
import connexion
import logging
from flask import request
logger = logging.getLogger(__name__)
conn_app = connexion.FlaskApp(__name__)
#conn_app.app.before_request
def before_request():
for h in request.headers:
logger.debug('header %s', h)
logger.debug('data %s', request.get_data())
I've been using requests.Session() to make web requests with authentication. Maybe 70% of the time I'll get a status_code of 200, but I also sporadically get 401.
Since I'm using a session - I'm absolutely positive that the credentials are correct - given that the same exact request when repeated may return 200.
Some further details:
I'm working with the SharePoint REST API
I'm using NTLM Authentication
To circumvent the problem, I've tried writing a loop that will sleep for a few seconds and retry the request. The odd thing here is that I haven't seen this actually recover - instead if the first request fails, then all subsequent requests will fail too. But if I just try again - the request may succeed on the first try.
Please note that I've already reviewed this question, but the suggestion is to use requests.Session(), which I'm already doing and still receiving 401s.
Here's some code to demonstrate what I've tried so far.
import requests
from requests_ntlm import HttpNtlmAuth
from urllib.parse import quote
# Establish requests session
s = requests.Session()
s.auth = HttpNtlmAuth(username, password)
# Update the request header to request JSON formatted output
s.headers.update({'Content-Type': 'application/json; odata=verbose',
'accept': 'application/json;odata=verbose'})
def RetryLoop(req, max_tries = 5):
''' Takes in a request object and will retry the request
upon failure up the the specified number of maximum
retries.
Used because error codes occasionally surface even though the
REST API call is formatted correctly. Exception returns status code
and text. Success returns request object.
Default max_tries = 5
'''
# Call fails sometimes - allow 5 retries
counter = 0
# Initialize loop
while True:
# Hit the URL
r = req
# Return request object on success
if r.status_code == 200:
return r
# If limit reached then raise exception
counter += 1
if counter == max_tries:
print(f"Failed to connect. \nError code = {r.status_code}\nError text: {r.text}")
# Message for failed retry
print(f'Failed request. Error code: {r.status_code}. Trying again...')
# Spacing out the requests in case of a connection problem
time.sleep(5)
r = RetryLoop(s.get("https://my_url.com"))
I've additionally tried creating a new session within the retry loop - but that hasn't seemed to help either. And I thought 5 seconds of sleep should be sufficient if it's a temporary block from the site, because I've retried myself in much less time and gotten the expected 200. I would expect to see a failure or two, and then a success.
Is there an underlying problem that I'm missing? And is there a more proper what that I can re-attempt the request given a 401?
** EDIT: #Swadeep pointed out the issue - by passing in the request to the function it's only calling the request once. Updated code that works properly:
def RetryLoop(req, max_tries = 5):
''' Takes in a request object and will retry the request
upon failure up the the specified number of maximum
retries.
Used because error codes occasionally surface even though the
REST API call is formatted correctly. Exception returns status code
and text. Success returns request object.
Default max_tries = 5
'''
# Call fails sometimes - allow 5 retries
counter = 0
# Initialize loop
while True:
# Return request object on success
if req.status_code == 200:
return req
# If limit reached then raise exception
counter += 1
if counter == max_tries:
print(f"Failed to connect. \nError code = {req.status_code}\nError text: {req.text}")
# Message for failed retry
print(f'Failed request. Error code: {req.status_code}. Trying again...')
# Spacing out the requests in case of a connection problem
time.sleep(1)
req = s.get(req.url)
This is what I propose.
import requests
from requests_ntlm import HttpNtlmAuth
from urllib.parse import quote
# Establish requests session
s = requests.Session()
s.auth = HttpNtlmAuth(username, password)
# Update the request header to request JSON formatted output
s.headers.update({'Content-Type': 'application/json; odata=verbose', 'accept': 'application/json;odata=verbose'})
def RetryLoop(s, max_tries = 5):
'''Takes in a request object and will retry the request
upon failure up the the specified number of maximum
retries.
Used because error codes occasionally surface even though the
REST API call is formatted correctly. Exception returns status code
and text. Success returns request object.
Default max_tries = 5
'''
# Call fails sometimes - allow 5 retries
counter = 0
# Initialize loop
while True:
# Hit the URL
r = s.get("https://my_url.com")
# Return request object on success
if r.status_code == 200:
return r
# If limit reached then raise exception
counter += 1
if counter == max_tries:
print(f"Failed to connect. \nError code = {r.status_code}\nError text: {r.text}")
# Message for failed retry
print(f'Failed request. Error code: {r.status_code}. Trying again...')
# Spacing out the requests in case of a connection problem
time.sleep(5)
r = RetryLoop(s)
I've tried to create my first telegram bot hosting the code as an amazon lambda instance, i suppose i should return something to the webhook 'cause it keep saying "Wrong response from the webhook: 502 Bad Gateway".
Here is part of my code:
def msgSend(text, chat_id):
url = URL + "sendMessage?text={}&chat_id={}".format(text, chat_id)
response = requests.get(url)
content = response.content.decode("utf8")
return content
def handle(msg):
sender = msg['from']['username']
id_gruppo = msg['chat']['id']
if sender == NAME:
testo = msg['text']
usernames = [x.replace('#','') for x in rx.findall(text)]
map(foo, usernames)
msgSend(confirm_mess, id_group)
return
def main(event, context):
response = ast.literal_eval(event['body'])
handle(response['message'])
return {
'something': 'something'
}
Actually the process works fine enough, the messages are received by my lambda and everything works like a charm, except for one thing, the confirmation message is sent over and over endlessly and the webhooks never marks the messages as read.
here is the response of getWebHookInfo:
{"ok":true,"result":{"url":"https://t2rt9guj3h.execute-api.us-west-2.amazonaws.com/prod/instabot","has_custom_certificate":false,"pending_update_count":19,"last_error_date":1489331750,"last_error_message":"Wrong response from the webhook: 502 Bad Gateway","max_connections":40}}
According to the bot helper the wh requires a 2XX code response...
Any idea about that?
According to the bot helper the wh requires a 2XX code response...
This is true. Your last statement should be
return {
statusCode: 200
}
If you don't return a successful response code, Telegram won't know what to do with it, which is why you see the HTTP 502 Bad Gateway. I was hitting this for awhile also :)
I have simple view like that:
#blueprint.route('/')
def index():
'''CMS splash page!'''
print request.form['no-key-like-that']
return render_template('home/splash.html', title='Welcome')
The goal of that view is to cause BadRequest error on Flask. That happens and I got very generic error page in the process that says:
Bad Request
The browser (or proxy) sent a request that this server could not understand.
However I want to intercept all errors and serve them wrapped in our templates for better look and feel, I do this like that:
#app.errorhandler(TimeoutException)
def handle_timeout(error):
if utils.misc.request_is_xhr(request):
return jsonify({'api_timeout': True}), 503
return redirect(url_for('errors.api_timeout', path=request.path))
However the same cannot be done for BadRequest exception, I tried:
#app.errorhandler(werkzeug.exceptions.BadRequestError)
def handle_bad_request(error):
return render_template(
'errors/bad_request.html',
exception_message=unicode(error),
return_path=request.path), 400
And:
#app.errorhandler(werkzeug.exceptions.BadRequestKeyError)
def handle_bad_request(error):
return render_template(
'errors/bad_request.html',
exception_message=unicode(error),
return_path=request.path), 400
And:
#app.errorhandler(werkzeug.exceptions.HttpError)
def handle_bad_request(error):
return render_template(
'errors/bad_request.html',
exception_message=unicode(error),
return_path=request.path), 400
In each case, instead of my bad_request.html there is raw response I mentioned above.
Bad Request
The browser (or proxy) sent a request that this server could not understand.
What actually works for me:
# NOTE: for some reason we cannot intercpet BadRequestKeyError, didn't
# figure out why. Thus I have added check if given 400 is
# BadRequestKeyError if so display standard api error page. This happens
# often when dev tries to access request.form that does not exist.
#app.errorhandler(400)
def handle_bad_request(error):
if isinstance(error, werkzeug.exceptions.BadRequestKeyError):
if utils.misc.request_is_xhr(request):
return jsonify({
'api_internal_error': True,
'error': unicode(error)
}), 500
return render_template(
'errors/api_internal.html',
exception_message=unicode(error),
return_path=request.path), 500
However as you can see it's far from perfection as error 400 is not necessarily always BadRequestKeyError.
Because exception handling works for any other exception but not BadRequest family it keeps me wondering, is it a bug? Or perhaps I am doing something wrong.