I have a Flask application on Google App Engine and I want to tell browsers to cache a response that uses the Cache-Control header. It works as expected on dev_appserver.py but when deployed to App Engine the headers are modified and break the cache header.
Here is the Flask view in particular:
#app.route("/resource")
def resource():
response = make_response(render_template("resource.html"))
response.headers['Cache-Control'] = "max-age=31536000"
logging.error("HEADERS: {}".format(response.headers))
return response
The logs for both development server and App Engine show:
Content-Type: text/html; charset=utf-8
Content-Length: 112628
Cache-Control: max-age=31536000
When I run it with the development app server it works as expected, as you can see from the headers below.
When I open Chrome's development tools the headers for App Engine are:
alternate-protocol:443:quic
cache-control:no-cache, must-revalidate
content-encoding:gzip
content-length:19520
content-type:text/html; charset=utf-8
date:Wed, 22 Jan 2014 19:53:47 GMT
expires:Fri, 01 Jan 1990 00:00:00 GMT
pragma:no-cache
server:Google Frontend
set-cookie:session=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
status:200 OK
strict-transport-security:max-age=31536000
vary:Accept-Encoding
version:HTTP/1.1
x-appengine-estimated-cpm-us-dollars:$0.002267
x-appengine-resource-usage:ms=7388 cpu_ms=5069
x-frame-options:DENY
x-ua-compatible:chrome=1
In contrast the development app server headers are as expected:
Cache-Control:max-age=31536000, private
Content-Length:112628
content-type:text/html; charset=utf-8
Date:Wed, 22 Jan 2014 19:57:05 GMT
Expires:Wed, 22 Jan 2014 19:57:05 GMT
Server:Development/2.0
set-cookie:session=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
x-frame-options:DENY
x-ua-compatible:chrome=1
Of course I have checked to make sure that I am not adding the extra headers, and I could find no reference of the cache-related headers (pragma, expires and cache-control) being added outside the given view.
So it seems App Engine is adding a bunch of headers when deployed, which seems unusual. What might I have overlooked?
-- EDIT --
As #dinoboff noted from the docs in a comment below:
Cache-Control, Expires and Vary
These headers specify caching policy to intermediate web proxies (such as Internet Service Providers) and browsers. If your script sets these headers, they will usually be unmodified, unless the response has a Set-Cookie header, or is generated for a user who is signed in using an administrator account.
These headers are additional headers that are added because you are looking at the site as a logged in admin user. They will not be present for "normal" users.
This blog post talks about the X-AppEngine-Resource-Usage header specifically: http://googleappengine.blogspot.co.uk/2009/08/new-features-in-124.html
And as they note:
You can view these headers using plugins such as Firefox's Live HTTP
Headers or Firebug. Note that only logged in administrators see these
figures - ordinary users, and users who aren't logged in, won't see
them at all.
Related
I am trying to enable caching for static files such as .css and .js.
I am running a WSGI server with Python.
I have tried setting the following headers for caching purposes:
headers.add_header('Cache-control', f'public, max-age={expires.strftime(RFC_1123_DATE)}')
headers.add_header('Expires', expires.strftime(RFC_1123_DATE))
headers.add_header('Last-Modified', generate_last_modified())
Headers recieved in the browser:
HTTP/1.0 200 OK
Date: Tue, 21 Apr 2020 08:06:17 GMT
Server: WSGIServer/0.2 CPython/3.6.9
Content-Encodings:
Content-Type: text/css; charset=UTF-8
Cache-control: public, max-age=Tue, 28 Apr 2020 08:06:17 GMT
Expires: Tue, 28 Apr 2020 08:06:17 GMT
Content-Length: 23399
Last-Modified: Tue, 21 Apr 2020 08:06:1587452777S GMT
Accept-Ranges: bytes
When using Chrome this code works and the files are stored and retrieved from the cache as you would expect. Chrome is using the Expires header and is ignoring the Cache-Control header.
I checked my developer tools and Disable Cache is not enabled. I checked my settings in Firefox in about:config, caching seems to be enabled.
So what am I missing here? Am I missing a header, is an ETAG required, why is Expires working in Chrome but not in Firefox?
I found the solution.
Firefox cache was full, so after emptying the cache it started sending the if-modified-since header again.
Also my server was returning the current time as the last-modified time instead of the actual last modified time.
To fix the issue all I had to do was compare the if-modified-since time from the browser with the last modified time from the file and send a 304 status if nothing changed.
I am trying to set a cookie in my browser using a Python Flask backend however, when we call the set cookie function I am unable to observe the cookie being saved by the browser. The following is my current understanding about how this is supposed to work:
Cookies are just key value pairs that may have an expiration which makes them persistent, otherwise they expire when the browser is closed
to set a cookie, all that is necessary is to use the set-cookie header in the response. I call the flask response object's set_cookie method to do this.
the browser should automatically save the cookie and follow the expiration rules (the set_cookie header can be observed in the response received by the browser)
Making the request in Angular HttpClient
let headers = new Headers();
headers.append('Content-Type', 'application/json');
let options = new RequestOptions({ headers: headers, withCredentials: true });
const request_data = {'username': this.username, 'password': this.password};
this.http.post('http://localhost:8080/token', request_data, options)
Setting the cookie in Python Flask
g.response = make_response()
time = datetime.datetime.now() + datetime.timedelta(days=30)
g.response.set_cookie("auth_token", auth.token, expires=time)
return g.response
Plain text response in the browser
HTTP/1.1 200 OK
set-cookie: auth_token=7253f2fa43d7584741dcf8972dea8f; Expires=Fri, 05-Jan-2018 01:33:30 GMT; Path=/
vary: Origin
access-control-allow-credentials: true
access-control-allow-origin: http://127.0.0.1:4200
content-type: application/json
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache
Content-Length: 58
Server: Development/2.0
Date: Wed, 06 Dec 2017 01:33:30 GMT
Browser's cookies section
Other thoughts & posts explored:
Tried using both Safari and Chrome, and received the same result in both. I have also verified that cookies are allowed by the browser.
$http doesn't send cookie in Requests
Cookie is not set in browser
How do I SET a Cookie (header) with XMLHttpRequest in JavaScript?
Question:
How do I get the cookies to be saved by the browser so that it can be used in the current session?
The domain for the cookie was set to the loopback address (127.0.0.1). In angular, I was calling the set-cookie endpoint using 'localhost' instead of the loopback address which prevented the cookies to be saved in the browser. As soon as cookie domain, endpoint base URL, and browser address matched using the loopback address, everything worked as expected.
Interesting side note: I am not sure why at the moment, but matching addresses doesn't seem to enough. I also tried setting both the cookie domain, endpoint base URL, and browser address to 'localhost' but this still didn't set the cookie. It only worked once all values were the loopback address.
So I am trying to make a script that checks a reservation availability of a bus. The starting link for this is https://reservation.pc.gc.ca/.
In the reserve box the following needs to be selected:
Reservation: Day Use (Guided Hikes, Lake O’Hara Bus)
Park: Yoho-Lake O'Hara
Arrival Date: Jun 16
Party Size: 2
When those options are entered, it takes you to the following page: https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar
It is my understanding that if I send a POST request to that second link with the correct data it should return the page I'm looking for
If I look in the dev tools network info when I select the correct parameters the form data is:
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE: -reallllly long string-
__VIEWSTATEGENERATOR: 8D0E13E6
ctl00$MainContentPlaceHolder$rdbListReservationType: Events
ddlLocations: 213a1bc9-9218-4e98-9a7f-0f209008e437**
ddlArrivalMonth: 2017-06-16
ddlArrivalDay: 19
ddlNights: 1
ddlDepartureMonth:
ddlDepartureDay:
ddlEquipment:
ddlEquipmentSub:
ddlPartySize:2
ctl00$MainContentPlaceHolder$chkExcludeAccessible: on
ctl00$MainContentPlaceHolder$imageButtonCalendar.x: 64
ctl00$MainContentPlaceHolder$imageButtonCalendar.y: 56
So the code I wrote is:
import requests
payload = {
'__EVENTTARGET': '',
'__EVENTARGUMENT': '',
'__VIEWSTATE':-reallly long string-,
'__VIEWSTATEGENERATOR': '8D0E13E6',
'ctl00$MainContentPlaceHolder$rdbListReservationType': 'Events',
'ddlLocations': '213a1bc9-9218-4e98-9a7f-0f209008e437',
'ddlArrivalMonth': 2017-06-16,
'ddlArrivalDay': 19,
'ddlNights': 1,
'ddlDepartureMonth': '',
'ddlDepartureDay': '',
'ddlEquipment': '',
'ddlEquipmentSub': '',
'ddlPartySize': 2,
'ctl00$MainContentPlaceHolder$chkExcludeAccessible': 'on',
'ctl00$MainContentPlaceHolder$imageButtonCalendar.x': 64,
'ctl00$MainContentPlaceHolder$imageButtonCalendar.y': 56
}
r = requests.get(r"https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar", data=payload)
print r.text
r.text ends up just being the second link as if no parameters were entered - as if I just sent a normal GET request to the link. I tried turning the payload values that are integers into strings, I tried removing the empty key:value pairs. No luck. Trying to figure out what I'm missing.
It looks to me like there are 2 things going on:
#errata was correct, and this should be a POST request. You're about halfway there.
What I noticed though is that it seems to post the form data to Home.aspx and the URL that you see after submitting the form is the result of that processing and subsequent redirect.
You might try posting the form data as json to ./Home.aspx.
I found through Postman that this nearly worked, but I had to specify the content-type in order to get the proper results.
If you need to know how to add header and body instructions to the .post() method, it looks like there is a good example (though perhaps slightly outdated) here:
adding header to python request module
Also, fwiw, check out Postman. If you're both inexperienced with requests and with doing it in Python, at least this may lesson some of the burden of trial and error.
You are using
r = requests.get(r"https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar", data=payload)
instead of
r = requests.post(r"https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar", data=payload)
Digging a bit deeper in your problem, I found out that the URL you are calling is actually redirecting to a different URL (returning HTTP response 302):
$ curl -I "https://reservation.pc.gc.ca/Yoho-LakeO'Hara"
HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 77273
Content-Type: text/html; charset=utf-8
Location: https://reservation-pc.fjgc-gccf.gc.ca/GccfLanguage.aspx?lang=eng&ret=https%3a%2f%2freservation.pc.gc.ca%3a443%2fYoho-LakeO%27Hara
Server: Microsoft-IIS/8.0
Set-Cookie: ASP.NET_SessionId=qw4p4e2zxjxx0c2zyq014p45; path=/; secure; HttpOnly
Set-Cookie: CookieLocaleName=en-CA; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Wed, 17 May 2017 14:22:53 GMT
However, following the Location from response results also in 302:
$ curl -I "https://reservation-pc.fjgc-gccf.gc.ca/GccfLanguage.aspx?lang=eng&ret=https%3a%2f%2freservation.pc.gc.ca%3a443%2fYoho-LakeO%27Hara"
HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 179
Content-Type: text/html; charset=utf-8
Location: https://reservation.pc.gc.ca:443/Yoho-LakeO'Hara?gccf=true
Server: Microsoft-IIS/8.0
Set-Cookie: ASP.NET_SessionId=rbcuvexfg4fb340ixtcjd1qy; path=/; secure; HttpOnly
Set-Cookie: _gc_lang=eng; domain=.fjgc-gccf.gc.ca; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Wed, 17 May 2017 14:24:55 GMT
All this probably results in Requests transforming your POST into GET in the end...
I tried using this
#app.after_request
def add_header(response):
response.headers['Cache-Control'] = 'max-age=300'
return response
But this causes a duplicate Cache-Control header to appear. I only want max-age=300, NOT the max-age=1209600 line!
$ curl -I http://my.url.here/
HTTP/1.1 200 OK
Date: Wed, 16 Apr 2014 14:24:22 GMT
Server: Apache
Cache-Control: max-age=300
Content-Length: 107993
Cache-Control: max-age=1209600
Expires: Wed, 30 Apr 2014 14:24:22 GMT
Content-Type: text/html; charset=utf-8
Use the response.cache_control object; this is a ResponseCacheControl() instance letting you set various cache attributes directly. Moreover, it'll make sure not to add duplicate headers if there is one there already.
#app.after_request
def add_header(response):
response.cache_control.max_age = 300
return response
You can set the default value for all static files when you create the Flask application:
app = Flask(__name__)
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 300
Note that if you modify request.cache_control in after_request, as in the accepted answer, this will also modify the Cache-Control header for static files and may override the behavior you set as I showed above. I'm currently using the following code to completely disable caching for dynamically generated content but not static files:
# No cacheing at all for API endpoints.
#app.after_request
def add_header(response):
# response.cache_control.no_store = True
if 'Cache-Control' not in response.headers:
response.headers['Cache-Control'] = 'no-store'
return response
Not completely sure this is the best way, but it's working for me so far.
I am testing my web-tornado application using Firefox's HttpRequest add-on but after I log in and receive my secure cookie data, I am not able to re-use it to consume protected methods.
This is my response data:
POST http://mylocalurl:8888/user/login
Content-Type: application/x-www-form-urlencoded
Login=mylogin;Pass=123
-- response -- 200 OK Content-Length: 33
Content-Type: text/html; charset=UTF-8
Server: TornadoServer/2.2.1
Set-Cookie:
IdUser="Mjk=|1395170421|ffaf0d6fecf2f91c0dccca7cab03d799ef6637a0";
expires=Thu, 17 Apr 2014 19:20:21 GMT; Path=/
{
"Success": true }
-- end response --
Now why I am trying to do is to configure HttpRequester to use this cookie for my new requests. I tried to add it using the "Headers" tab but my server keeps sending me a 403, Forbidden.
Can anyone help me on this ? It could be with another tool (for linux) too.
I really like fiddler2 for these kind of things and there's an alpha build for mono that you may wish to try out: http://www.telerik.com/download/fiddler
If you don't mind paid software you can use Charles, for which there is a free trial.
And if you are testing and already using python, why not use a simple python script with requests and its Session object with cookie-persistence..