We have our own private cloud where I have installed OpenStack Swift. I have a working node (proxy and storage) that allows me to store and retrieve if I use the openstack and swift python cli to store and retrieve files. Additionally I am able to use the python API on a remote machine to store and retrieve files.
The root of my question is how to debug temp url and crossdomain filter issues. Is there a way to turn on detailed debug logging for these filters?
I have the default logging set to
log_name = swift
log_facility = LOG_LOCAL0
log_level = DEBUG
The situation I am trying to troubleshoot is as follows. When I try and use temp url and cross domain (for CORS), I get a 401. I debugged the code and it appears to be a invalid HMAC error. Based on research, this appears to be a date time issue where the client and the server have missed matched times. However both are running the ntpd service so the time should be in sync.
For CORS, it appears that preflight OPTIONS request is succeeding. The subsequent PUT is failing with a 401....
"No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://blahost' is therefore not allowed access. The response had HTTP status code 401."
The strange part is the OPTIONS request is returning "access-control-allow-origin" instead of 'Access-Control-Allow-Origin'... the case is off.
Preflight request:
OPTIONS /v1/AUTH_99cf99f26aaa4b2c923806231b03334c/436/88b6d895-6dbf-4f29-904d-96c9b7959016?temp_url_sig=4a953c34372e37b2a22bb31fb0581a7eb7f02cee&temp_url_expires=1441508891 HTTP/1.1
Host: 23.253.200.41:8080
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: PUT
Origin: http://blahhost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Access-Control-Request-Headers: accept, content-type
Accept: */*
Referer: http://blahost/binder/436/site/419/folder/17560/file
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Preflight Response:
HTTP/1.1 200 OK
access-control-allow-origin: http://blahost
access-control-allow-methods: HEAD, GET, PUT, POST, COPY, OPTIONS, DELETE
access-control-allow-headers: content-type, accept
Allow: HEAD, GET, PUT, POST, COPY, OPTIONS, DELETE
Content-Length: 0
X-Trans-Id: tx9e359777dfb94148858cd-0055eba012
Date: Sun, 06 Sep 2015 02:08:18 GMT
Connection: keep-alive
Subsequent PUT request(notice it is missing the Access-Control-Allow-Origin)
PUT /v1/AUTH_99cf99f26aaa4b2c923806231b03334c/436/88b6d895-6dbf-4f29-904d-96c9b7959016?temp_url_sig=4a953c34372e37b2a22bb31fb0581a7eb7f02cee&temp_url_expires=1441508891 HTTP/1.1
Host: 23.253.200.41:8080
Connection: keep-alive
Content-Length: 16231
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
Origin: http://blahost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Content-Type: application/pdf
Referer: http://blahost/binder/436/site/419/folder/17560/file
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I would appreciate any advice on how to troubleshoot.
Thanks
Greg
Related
On the instagram login page, if one inspects the element of the POST call for the url 'https://www.instagram.com/accounts/web_create_ajax/', it lists the following as headers:
Host: www.instagram.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.instagram.com/
X-CSRFToken: 7dmO9F3JuVGvSXumd79yByPxnHoWHz1A
X-Instagram-AJAX: c2d8f4380025
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Content-Length: 102
Cookie: csrftoken=7dmO9F3JuVGvSXumd79yByPxnHoWHz1A; mid=W30zsQAEAAErXHJ3iUojfTceCd53; mcd=3; csrftoken=7dmO9F3JuVGvSXumd79yByPxnHoWHz1A; rur=FTW
Connection: keep-alive
I am wondering if anyone would have any idea what X-Instagram-AJAX is and how I can generate it each time. Is it connected as a pair with X-CSRFToken? Thanks.
Follow, like etc requests working without this header. I don't know what is it but i think instagram dedects suspicious requests with this and then log it. You can get this value on any page in instagram This is x-instagram-ajax value
You can parse it and use.
I've got problem while trying to post the file to the server. I'm trying to make file upload script to server, this server is very 'Sensitive to correctness post request'
I debugged page that is sending the file to server and browser send this (TextView):
POST http://example.com/post HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 20625
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarykGHBkXoER9gNuVna
Referer: http://example.com/foo
Accept-Encoding: gzip, deflate
Accept-Language: pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4,pt;q=0.2
------WebKitFormBoundarykGHBkXoER9gNuVna
Content-Disposition: form-data; name="files[]"; filename="file.zip"
Content-Type: application/octet-stream
...raw file data...
------WebKitFormBoundarykGHBkXoER9gNuVna--
However, my script is sending this (TextView):
POST http://example.com/post HTTP/1.1
Host: example.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.18.1
Content-Length: 20604
--f8c266cf436941019c5a80c7d4779a57
Content-Disposition: form-data; name="files[]"; filename="file.zip"
Content-Type: application/zip
...raw file data...
--f8c266cf436941019c5a80c7d4779a57--
With causes error on server, additional note: this error started when I changed files=files to data=files
Current Code:
files = MultipartEncoder({'files[]': (filename, open(local_path,'rb'), mimetype)})
UploadFile = requests.post(self.UploadURL, data=files, allow_redirects=False)
Working code:
files = {'files[]': (filename, open(local_path,'rb'), mimetype)}
UploadFile = requests.post(self.UploadURL, files=files, allow_redirects=False)
I'm using MultipartEncoder to allow sending huge files.
I see that biggest mismatch is "boundary", but why this 'boundary' is generating in working code but in Current code not?
How to fix that?
You are not setting the Content-Type header, the MultipartEncoder provides it for you:
files = MultipartEncoder({'files[]': (filename, open(local_path,'rb'), mimetype)})
UploadFile = requests.post(
self.UploadURL, data=files, allow_redirects=False,
headers={'Content-Type': files.content_type})
The header must come from the multi-part encoding, because it is responsible for picking the boundary used to deliniate the various MIME parts in the multipart response. In your upload that's:
--f8c266cf436941019c5a80c7d4779a57
but it is generated at random each time your code runs. The header provided would look like:
Content-Type: multipart/form-data; boundary=--f8c266cf436941019c5a80c7d4779a57
I am trying to log into my router's panel using python, but the problem is that I have no idea what the protocol for doing that is. I tried using Wireshark to find out, but it just shows just a GET request and a response. I tried logging in to the router and then searching the username and password in the packets, but it didn't find it. (My guess is that it's encrypted)
If anyone could help me with the protocol of logging in to the panel, it would be greatly appreciated.
Found it. Fllowing the TCP stream gave me the following:
GET / HTTP/1.1
Host: 10.0.0.138
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Basic UG90YXRvOg==
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,he;q=0.6
HTTP/1.0 401 Unauthorized
WWW-Authenticate: Basic realm="NETGEAR DGN2200v2BEZEQ"
Content-type: text/html
<html>
<head><title>401 Unauthorized</title></head>
<body><h1>401 Unauthorized</h1>
<p>Access to this resource is denied, your client has not supplied the correct authentication.</p></body>
</html>
The username and password are encoded in base64 in the format of username:password.
I'm trying to get single sign-on working in Azure Active Directory, using this bit of documentation as a guide. However, when I get up to the "access token request" stage, I get the following error:
Error validating credentials. AADSTS70000: The provided access grant is invalid or malformed.
Searching the Internet, it seems that this is usually caused by the redirect_uri parameter being missing in the second step or different between the two steps, but that doesn't seem to be the case here.
Here's what's happening in each step:
Authorize step (raw HTTP request from browser):
GET /[snip tenant id]/oauth2/authorize?redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Fpost-login%3Fdest%3D%252F&response_type=code&client_id=[snip client id] HTTP/1.1
Host: login.windows.net
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-AU,en;q=0.8,en-US;q=0.6
Cookie: [snip a handful of cookies]
Redirect step (raw HTTP request from browser):
GET /post-login?code=[snip base64]&session_state=[snip uuid] HTTP/1.1
Host: localhost:5000
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-AU,en;q=0.8,en-US;q=0.6
Cookie: csrftoken=vBjLMAFTw7NSFEJHb2t9GTA0Eoced4rw; azure-redirect-uri="http://localhost:5000/post-login?dest=%2F"
Token request step (raw HTTP request from server code):
POST /[snip tenant id]/oauth2/token HTTP/1.1
Host: login.windows.net
Content-Length: 805
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.5.0 CPython/2.7.6 Darwin/14.0.0
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Fpost-login%3Fdest%3D%252F&client_secret=Dsysz7F%2FXh2Wu1YKE%2BVEOkvMHhvc38DnwFTa5qekyXM%3D&code=[snip base64]&client_id=[snip client id]&grant_type=authorization_code
Response to token request (Python dict parsed from JSON returned):
{
u'timestamp': u'2014-12-09 05:37:58Z',
u'trace_id': u'ae00a782-30f1-4e1c-a183-f19330ecca37',
u'submit_url': None,
u'correlation_id': u'21a7b861-5171-4083-9da5-67e7d956ab5e',
u'error_description': u'AADSTS70002: Error validating credentials. AADSTS70000: The provided access grant is invalid or malformed.\r\nTrace ID: ae00a782-30f1-4e1c-a183-f19330ecca37\r\nCorrelation ID: 21a7b861-5171-4083-9da5-67e7d956ab5e\r\nTimestamp: 2014-12-09 05:37:58Z',
u'context': None,
u'error': u'invalid_grant',
u'error_codes': [70002, 70000]
}
Your token request is failing because of the query parameter at the end of your redirect uri. OAuth redirect URI's should not have any query parameters or fragments. You can use the state parameter instead.
I don't think that the authorize call should have succeeded either, but clearly it did. I am checking to see if that is a bug.
See this link for some more information on how to use the OAuth state parameter:
http://www.thread-safe.com/2014/05/the-correct-use-of-state-parameter-in.html
I've followed the following article in an attempt to setup Apache2 caching in order to use it with Django on Ubuntu 12.10 with mod_wsgi. I want Apache to cache some requests for me.
http://www.howtoforge.com/caching-with-apaches-mod_cache-on-ubuntu-10.04
From the article I enabled the modules and setup the following php script to test the caching. The caching works just fine - I only get a new timestamp after 5 minutes.
vi /var/www/cachetest.php
<?php
header("Cache-Control: must-revalidate, max-age=300");
header("Vary: Accept-Encoding");
echo time()."<br>";
?>
Now in my django response, I return an HttpResponse object after setting the appropriate headers the same way:
# Create a Response Object with the content to return and set it's
response = HttpResponse("%s"%(output_display))
response['Cache-Control'] = 'must-revalidate, max-age=20'
response['Vary'] = 'Accept-Encoding'
return response
The caching with the Django request doesn't work at all. I've used Firefox's LiveHeaders to examine the HTTP response headers.
For the example link above and the PHP script the headers look like:
http://localhost/cachetest.php
GET /cachetest.php HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cache-Control: max-age=0
HTTP/1.1 200 OK
Date: Sun, 10 Mar 2013 02:29:32 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.4.6-1ubuntu1.1
Cache-Control: must-revalidate, max-age=300
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 34
Connection: close
Content-Type: text/html
----------------------------------------------------------
For my Django Request - the caching doesn't work, it always forces the lengthy operation to complete the response - just like re-loading the php request above with F5. Using the FireFox plugin I seem to be writing the correct headers:
http://localhost/testdjango/testdjango/
GET /testdjango/testdjango/ HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Date: Sun, 10 Mar 2013 02:32:41 GMT
Server: Apache/2.2.22 (Ubuntu)
Vary: Accept-Encoding
Cache-Control: must-revalidate, max-age=20
Content-Encoding: gzip
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
----------------------------------------------------------
What am I doing wrong? How can I get the django caching to work like the php script? Thanks!
This seems to be your problem:
Transfer-Encoding: chunked
It means a 'streaming response', in terms of mod_mem_cache. And, according to the docs:
By default, a streamed response will not be cached unless it has a
Content-Length header.
You can solve it by setting the MCacheMaxStreamingBuffer directive.