In my Google App Engine application, I have a handler with a method for PUT requests:
def postMethod(self, arg):
response = do_backend_work(arg)
if response.field is None:
self.error(502)
self.response.out.write(json.dumps(
{'message': "you've been a bad boy!"}))
else:
<deal with well-formatted requests here>
.
.
.
However, when I do receive a request where response.field is None the request is returned as a 200. When I caught this error I inserted raise Exception(str(self.response)) just before the else block as a sanity check and found this in the logs:
raise Exception(str(self.response))
Exception: 502 Bad Gateway
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Length: 63
{"message": "you've been a bad boy!"}
INFO 2012-08-15 23:25:12,239 dev_appserver.py:2952]
"PUT /url/resource HTTP/1.1" 200 -
is there something I'm missing about how App Engine is processing the request?
After some rooting around, I figured out the problem. in my do_backend_work function, there was a rogue print statement which was hijacking the output stream! The moral of the story is: don't write to stdout on your Google App Engine back-end. I'm posting the question for the benefit of future generations.
Related
I am trying to connect to a Confluence page using the python wrapper on the API (as I am not familiar with any of this) but I keep getting the following error:
requests.exceptions.HTTPError: 401 Client Error
I know that people talk about this being caused by the necessity of using an API token but the page runs on an old version of Confluence and I have been told that we cannot use access tokens.
So has anyone any other idea? Here's a small code:
from atlassian import Confluence
confluence = Confluence(
url='https://address',
username='name',
password='pwd'
)
confluence.create_page(
space='Test',
title='A title',
body='something')
I have tried to use an older version of atlassian-python-api just in case there was some conflict but it got me the same error.
Your code looks ok. Authenticating to Confluence using Basic Auth should work without generating an API token, afaik.
The 401 status definitely suggests a problem with the authentication though. The obvious reason for this would be of course wrong credentials, but I assume that you have double checked that the credentials work when interactively logging into confluence with a browser.
To get a better sense of the error, you can import logging to debug your requests and response:
from atlassian import Confluence
import logging
logging.basicConfig(filename='conf_connect.log', filemode='w', level=logging.DEBUG)
try:
c = Confluence(url='https://conf.yoursystem.com', username='name', password='pwd')
# atlassian API does not raise error on init if credentials are wrong, this only happens on the first request
c.get_user_details_by_username('name')
except Exception as e:
logging.error(e)
The Confluence module internally also uses logging, so the requests and responses will appear in your conf_connect.log logfile:
DEBUG:atlassian.rest_client:curl --silent -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' 'https://conf.yoursystem.com/rest/api/user?username=name'
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conf.yoursystem.com:443
DEBUG:urllib3.connectionpool:https://conf.yoursystem.com:443 "GET /rest/api/user?username=name HTTP/1.1" 401 751
DEBUG:atlassian.rest_client:HTTP: GET rest/api/user -> 401
DEBUG:atlassian.rest_client:HTTP: Response text -> <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Basic Authentication Failure - Reason : AUTHENTICATED_FAILED</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.33</h3></body></html>
ERROR:root:401 Client Error: for url: https://conf.yoursystem.com/rest/api/user?username=name
The response body may include some information on the reason:
HTTP Status 401 – UnauthorizedType Status ReportMessage Basic Authentication Failure - Reason : AUTHENTICATED_FAILEDDescription The request has not been applied because it lacks valid authentication credentials for the target resource.
The reason AUTHENTICATED_FAILED suggests something is likely wrong with your credentials. If you want to dig deeper into that, you can use this SO answer to also display the headers that are being sent with your request.
However, if your reason is AUTHENTICATION_DENIED the problem is likely the following: If you have too many failed authentication attempts in a row, a CAPTCHA challenge is triggered, and this error will occur until the Failed Login Count is reset. This can easily happen when you are developing a script and test it frequently. To remedy this, either open a browser and manually (re-)logon to Confluence, completing the CAPTCHA, or resolve it from the Confluence User Management.
I've split my previous question into two.
A consumer of my REST API says that on occasion I am returning a 400 Bad Request - The request sent by the client was syntactically incorrect. error.
My application (Python/Flask) logs don't seem to be capturing this, and neither do my webserver/Nginx logs.
I've changed the default 400 and 404 error message HTML's by adding the following line to the config (and adding the corresponding HTML pages in the proper directory):
error_page 404 /404.html;
location = /404.html {
root html;
}
error_page 400 /400.html;
location = /400.html {
root html;
}
This error message plainly states "Nginx" so that I know Nginx is giving the 400 and not my python/flask application (whose 400 has been changed to plainly state "Flask").
Is there a way that I can intentionally cause Nginx to return a 400 error, perhaps through python, CURL, or Postman? For example, I've also changed the 404 error page, and can intentionally get Nginx to return the corresponding error HTML by calling an invalid URL.
Create a location in NGINX's config to return 400 errors:
location /error {
return 400;
}
Presto.
Nginx will return status code 400 if it receives a header field larger than the configured large_client_header_buffers
A request header field cannot exceed the size of one buffer as well,
or the 400 (Bad Request) error is returned to the client. Buffers are
allocated only on demand. By default, the buffer size is equal to 8K
bytes. If after the end of request processing a connection is
transitioned into the keep-alive state, these buffers are released.
Source: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers
So you just need to create a curl request with a header larger than 8k. Here's an example using a bit of python to generate the header variable to pass into curl:
(nginx)macbook:nginx joeyoung$ myheader=$(python -c "print 'A'*9000")
(nginx)macbook:nginx joeyoung$ curl -vvv --header "X-MyHeader: $myheader" http://my.example.website.com
Results:
...
>
< HTTP/1.1 400 Bad Request
< Server: nginx/1.4.7
< Date: Wed, 02 Sep 2015 22:37:29 GMT
< Content-Type: text/html
< Content-Length: 248
< Connection: close
I wrote my own custom client which sends raw http requests via my wifi card to my flask webserver.
This is what a typical requests looks like:
Content-Length: 214
User-Agent: blah
Connection: close
Host: 1.2.3.4:5000
Content-Type: application/json
{"data":[{"scoutId":2,"message":"ph=5.65"},{"scoutId":4,"message":"ph=4.28"},{"scoutId":3,"message":"ph=4.28"},{"scoutId":2,"message":"ph=5.65"},{"scoutId":4,"message":"ph=4.28"},{"scoutId":3,"message":"ph=4.30"}]}
Sometimes, my clients screw up and send malformed JSON requests to my flask server. Typically, flask will just display:
1.2.3.5 - - [01/Sep/2014 22:13:03] "POST / HTTP/1.1" 400 -
and nothing informative about the request.
I would like to track every single request that resulted in 400 in my environment and analyze what is causing these errors.
Where can I place my custom error function in my flask server?
Try turning this on:
app.config['TRAP_BAD_REQUEST_ERRORS'] = True
This should make flask raise an exception instead of just logging the 400 (see documentation here).
If you need to do something more than that, make an event handler:
http://flask.pocoo.org/docs/0.10/patterns/errorpages/
#app.errorhandler(400)
def page_not_found(exc):
#do something with the exception object `exc` here
....
Or try wrapping the body of your view function in try/except.
I'm trying to serve a blob from a blobKey that I've stored in a db.Model as a blobstore.BlobReferenceProperty() and I'm unsure of how to to this properly as my current method gives a 404. I am storing the blobKey value by taking the 'rewritten' request using the regular self.request.get('file_field')
the contents of the key look like this in the SDK console:
Content-Type: video/mp4
MIME-Version: 1.0
Content-Length: 475712
Content-MD5: OTY0MjY4OGI4NDgwYzVlZTI2MGJiNzg0YTA4OTIzNzY=
content-type: video/mp4
content-disposition: form-data; name="video_file"; filename="test_time.mp4"
X-AppEngine-Upload-Creation: 2012-11-10 21:41:12.973934
My video download handler is very standard and looks like this
class VideoServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)
Here's how i render the url:
'/videos/%s' % time_data_instance.video_key.key()
And the url mapping:
('/videos/([^/]+)?', VideoServeHandler)
And heres the 404 error i get from the console:
INFO 2012-11-10 21:54:11,371 dev_appserver.py:3092] "GET /videos/Content-Type:%20video/mp4MIME-Version:%201.0Content-Length:%20475712Content-MD5:%20OTY0MjY4OGI4NDgwYzVlZTI2MGJiNzg0YTA4OTIzNzY=content-type:%20video/mp4content-disposition:%20form-data;%20name= HTTP/1.1" 404 -
Does anybody know what the problem might be, looking at the url in the 404 its looks as if that is incorrect but i cant find any othehr way to generate it
Update:
Heres the entire code
http://www.pastebucket.com/5163
send_blob needs the BlobKey, not the BlobInfo.
Just pass resource to send_blob
class VideoServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
self.send_blob(resource)
The problem is with your request string:
INFO 2012-11-10 21:54:11,371 dev_appserver.py:3092] "GET /videos/Content-Type:%20video/mp4MIME-Version:%201.0Content-Length:%20475712Content-MD5:%20OTY0MjY4OGI4NDgwYzVlZTI2MGJiNzg0YTA4OTIzNzY=content-type:%20video/mp4content-disposition:%20form-data;%20name= HTTP/1.1" 404 -
As far as I can see you request /videos/Content-Type:%20video/mp4[..some garbage...], no wonder you get a 404 back.
Check again how you make a request URL. Definitely it does not look like this:
'/videos/%s' % time_data_instance.video_key.key()
I implemented the experimental OAuth support for Google App Engine using Python, and have it working locally, but the endpoints are throwing a 400 when I deploy to appspot.
For example, the url http(s)://my-app.appspot.com/_ah/OAuthGetRequestToken returns a 400, but locally that url pattern behaves as expected.
I have tried both http and https, and assumed that appspot handles the ssl cert.
UPDATE
I've been using the OAuth Playground to test my code. Despite documentation, it seems Registering your app is required. Go here for instructions on how to register. According to documentation during the registration process, certificate is not required when running on App Engine. Playground is showing more detail on the error - "signature invalid". If I understand correctly, the signature is produced from a signature base string. In this case I am using the base string 7DYB6MJ2s-IQcd7VJYJUmcct .
GET /accounts/OAuthGetRequestToken?scope=https%3A%2F%2Fmail.google.com%2Fmail%2Ffeed%2Fatom HTTP/1.1
Host: www.google.com
Accept: */*
Authorization: OAuth oauth_version="1.0", oauth_nonce="168cfd60a93a46caa38dddfdcedd9de9", oauth_timestamp="1305315895", oauth_consumer_key="xxxxxxx.appspot.com", oauth_callback="http%3A%2F%2Fgooglecodesamples.com%2Foauth_playground%2Findex.php", oauth_signature_method="HMAC-SHA1", oauth_signature="4J5faUujE0VNaybyvFCiEPY7DQ8%3D"
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=UTF-8
Date: Fri, 13 May 2011 19:44:55 GMT
Expires: Fri, 13 May 2011 19:44:55 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Length: 451
Server: GSE
**signature_invalid**
base_string:GET&https%3A%2F%2Fwww.google.com%2Faccounts%2FOAuthGetRequestToken&oauth_callback%3Dhttp%253A%252F%252Fgooglecodesamples.com%252Foauth_playground%252Findex.php%26oauth_consumer_key%3Dxxxxxx.appspot.com%26oauth_nonce%3D168cf60a94caa38e2defdcedd9de9%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1305315895%26oauth_version%3D1.0%26scope%3Dhttps%253A%252F%252Fmail.google.com%252Fmail%252Ffeed%252Fatom
FINAL UPDATE
There were two things causing the 400. First, the app was not registered. Google's documentation says it's optional, but that is not the case apparently. Secondly, the Request was not properly signed. Here is an excellent debugging tool to test your OAuth Requests: Oauth Playground
You must register your domain in order to have OAuth working on production.
Although the following docs state that Registering is Optional:
http://code.google.com/apis/accounts/docs/RegistrationForWebAppsAuto.html
It doesn't work without the Registration since January.
Look at the link above and Register your domain/application. You don't need to submit a certificate, this is still optional.
Which request method it is? In dev_appserver_oauth.py I see:
if method != 'GET' and method != 'POST':
outfile.write('Status: 400\r\n')
return
So it'll only work for GET or POST requests.
First ensure you have enabled Federated Login in your Application Settings.
From your description it sounds like you might just be performing a direct GET request to /_ah/OAuthGetRequestToken without any of the other required parameters of oAuth. This will work on the dev_appserver as it is simply a mockup of oAuth to let you flesh out your code.
See the parameters listed on the OAuthGetRequestToken description page for what is required and how to deal with signing. I believe you can ignore scope for GAE though