I am trying to set a cookie in my browser using a Python Flask backend however, when we call the set cookie function I am unable to observe the cookie being saved by the browser. The following is my current understanding about how this is supposed to work:
Cookies are just key value pairs that may have an expiration which makes them persistent, otherwise they expire when the browser is closed
to set a cookie, all that is necessary is to use the set-cookie header in the response. I call the flask response object's set_cookie method to do this.
the browser should automatically save the cookie and follow the expiration rules (the set_cookie header can be observed in the response received by the browser)
Making the request in Angular HttpClient
let headers = new Headers();
headers.append('Content-Type', 'application/json');
let options = new RequestOptions({ headers: headers, withCredentials: true });
const request_data = {'username': this.username, 'password': this.password};
this.http.post('http://localhost:8080/token', request_data, options)
Setting the cookie in Python Flask
g.response = make_response()
time = datetime.datetime.now() + datetime.timedelta(days=30)
g.response.set_cookie("auth_token", auth.token, expires=time)
return g.response
Plain text response in the browser
HTTP/1.1 200 OK
set-cookie: auth_token=7253f2fa43d7584741dcf8972dea8f; Expires=Fri, 05-Jan-2018 01:33:30 GMT; Path=/
vary: Origin
access-control-allow-credentials: true
access-control-allow-origin: http://127.0.0.1:4200
content-type: application/json
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache
Content-Length: 58
Server: Development/2.0
Date: Wed, 06 Dec 2017 01:33:30 GMT
Browser's cookies section
Other thoughts & posts explored:
Tried using both Safari and Chrome, and received the same result in both. I have also verified that cookies are allowed by the browser.
$http doesn't send cookie in Requests
Cookie is not set in browser
How do I SET a Cookie (header) with XMLHttpRequest in JavaScript?
Question:
How do I get the cookies to be saved by the browser so that it can be used in the current session?
The domain for the cookie was set to the loopback address (127.0.0.1). In angular, I was calling the set-cookie endpoint using 'localhost' instead of the loopback address which prevented the cookies to be saved in the browser. As soon as cookie domain, endpoint base URL, and browser address matched using the loopback address, everything worked as expected.
Interesting side note: I am not sure why at the moment, but matching addresses doesn't seem to enough. I also tried setting both the cookie domain, endpoint base URL, and browser address to 'localhost' but this still didn't set the cookie. It only worked once all values were the loopback address.
Related
Hi i am trying to access one rest api with can be accessed only after login. I was using below code but getting 401, access denied. I am sure if same cookies will be applied to next put call, it will not give access denied. but python session is not using the same cookies.. instead adding new cookies..thanks..
with requests.Session() as s:
logging.info("Trying to login")
response1 = s.post("https://localhost:8080/api/authentication?j_username=admin&j_password=admin", verify=False)
for cookie in s.cookies:
logging.info(str(cookie.name) + " : " + str(cookie.value))
logging.info("logged in successfully " + str(response1.status_code))
url = url1 % (params['key'])
logging.info("inspector profile inpect api : " + url)
response = s.put(url, verify=False)
for cookie in s.cookies:
logging.info(str(cookie.name) + " :: " + str(cookie.value))
logging.info("code:-->"+ str(response.status_code))
Output is
CSRF-TOKEN : c3ea875b-3df9-4bd4-992e-2b976c150ea6
JSESSIONID : M3WWdp0PO95ENQSJciqiEbiHZR6ge7O8HkKDkY6R
logged in successfully 200
profile api : localhost:8080/api/test/283
CSRF-TOKEN :--> e5b64a66-5402-430b-8f51-d8d7549fd84e
JSESSIONID :--> JUZBHKmqsitvlrPvWuaqfTJNH1PIJcEXPTkPYPKk
CSRF-TOKEN :--> c3ea875b-3df9-4bd4-992e-2b976c150ea6
JSESSIONID :--> M3WWdp0PO95ENQSJciqiEbiHZR6ge7O8HkKDkY6R
code:401
Looks like next api call is not using the cookies, please help me out.
Just finished debugging the same issue.
By RFC 2965:
The term effective host name is related to host name. If a host name
contains no dots, the effective host name is that name with the
string .local appended to it. Otherwise the effective host name is
the same as the host name. Note that all effective host names
contain at least one dot.
Python Requests module uses http.cookiejar module to handle the cookies. It verifies the received cookies before applying them to a session.
Use the following code to get debug output:
import logging
import http.cookiejar
logging.basicConfig(level=logging.DEBUG)
http.cookiejar.debug = True
Here is an example, when received cookie is not applied:
DEBUG:http.cookiejar:add_cookie_header
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST /api/login HTTP/1.1" 200 6157
DEBUG:http.cookiejar:extract_cookies: Date: Thu, 30 Apr 2020 15:45:11 GMT
Server: Werkzeug/0.14.1 Python/3.5.3
Content-Type: application/json
Content-Length: 6157
Set-Cookie: token=1234; Domain=localhost; Path=/
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
DEBUG:http.cookiejar: - checking cookie token=1234
DEBUG:http.cookiejar: non-local domain .localhost contains no embedded dot
Requests sent to localhost, expect web server to set domain part of a cookie to localhost.local
Here is an example, when received cookie was applied correctly:
DEBUG:http.cookiejar:add_cookie_header
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST /api/login HTTP/1.1" 200 6157
DEBUG:http.cookiejar:extract_cookies: Date: Thu, 30 Apr 2020 15:52:08 GMT
Server: Werkzeug/0.14.1 Python/3.5.3
Content-Type: application/json
Content-Length: 6157
Set-Cookie: token=1234; Domain=localhost.local; Path=/
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
DEBUG:http.cookiejar: - checking cookie token=1234
DEBUG:http.cookiejar: setting cookie: <Cookie token=1234 for .localhost.local/>
If you cannot fix the web server, use 127.0.0.1 instead of localhost in your request:
response1 = s.post("https://127.0.0.1:8080/api/authentication?j_username=admin&j_password=admin", verify=False)
This code worked for me:
from requests import Session
s = Session()
s.auth = ('username', 'password')
s.get('http://host'+'/login/page/')
response = s.get('http://host'+'/login-required-pages/')
You did not actually authenticate successfully to the website despite having CSRF-TOKEN and JSESSIONID cookies. The session data, including whether or not you're authenticated, are stored on the server side, and those cookies you're getting are only keys to such session data.
One problem I see with the way you're authenticating is that you're posting username and password as query string, which is usually only for GET requests.
Try posting with proper payload instead:
response1 = s.post("https://localhost:8080/api/authentication", data={'j_username': 'admin', 'j_password': 'admin'}, verify=False)
I'm stuck in web scraping a page using Python. Basically, the following is the request from HttpRequester (in Mozilla) and it gives me the right response.
POST https://www.hpe.com/h20195/v2/Library.aspx/LoadMore
Content-Type: application/json
{"sort": "csdisplayorder", "hdnOffset": "1", "uniqueRequestId": "d6da6a30bdeb4d77b0e607a6b688de1e", "test": "", "titleSearch": "false", "facets": "wildcatsearchcategory#HPE,cshierarchycategory#No,csdocumenttype#41,csproducttype#18964"}
-- response --
200 OK
Cache-Control: private, max-age=0
Content-Length: 13701
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 28 May 2016 04:12:57 GMT
Connection: keep-alive
The exact same operation in python 2.7.1 using Requests, fails with an error. The following is the code snippet:
jsonContent = {"sort": "csdisplayorder", "hdnOffset": "1", "uniqueRequestId": "d6da6a30bdeb4d77b0e607a6b688de1e", "test": "", "titleSearch": "false", "facets": "wildcatsearchcategory#HPE,cshierarchycategory#No,csdocumenttype#41,csproducttype#18964"}
catResponse = requests.post('https://www.hpe.com/h20195/v2/Library.aspx/LoadMore', json = jsonContent)
The following is the error that I get:
{"Message":"Value cannot be null.\r\nParameter name: source","StackTrace":" at
System.Linq.Enumerable.Contains[TSource](IEnumerable`1 source, TSource value, I
EqualityComparer`1 comparer)\r\n
More information:
The Post request that I'm looking for is fired upon:
opening this web page: https://www.hpe.com/h20195/v2/Library.aspx?doctype=41&doccompany=HPE&footer=41&filter_doctype=no&filter_doclang=no&country=&filter_country=no&cc=us&lc=en&status=A&filter_status=rw#doctype-41&doccompany-HPE&prodtype_oid-18964&status-a&sortorder-csdisplayorder&teasers-off&isRetired-false&isRHParentNode-false&titleCheck-false
Clicking on the "Load more" grey button at the end of the page
I'm capturing the exact set of request headers and response from the browser operation and trying to mimic that in Postman, Python code and HttpRequester (Mozilla).
It flags the same error (mentioned above) with Postman and Python, but works with no headers set on my part with HttpRequester.
Can anyone think of an explanation for this?
If both Postman and requests are receiving an error, then there is more context than what HttpRequester is showing. There are a number of headers that I'd expect to be set almost always, including User-Agent and Content-Length, that are missing here.
The usual suspects are cookies (look for Set-Cookie headers in earlier requests, preserve those by using a requests.Session() object), the User-Agent header and perhaps a Referrer header, but do look for other headers like anything starting with Accept, for example.
Have HttpRequester post to http://httpbin.org/post instead for example, and inspect the returned JSON, which tells you what headers were sent. This won't include cookies (those are domain-specific), but anything else could potentially be something the server looks for. Try such headers one by one if cookies are not helping.
I've ran this queries millions (yes, millions) of times before with other URLs. However, I'm getting a KeyError when checking the content-type of the following webpage.
Code snippet:
r = requests.get("http://health.usnews.com/health-news/articles/2014/10/15/limiting-malpractice-claims-may-not-curb-costly-medical-tests", timeout=10, headers=headers)
if "text/html" in r.headers["content-type"]:
Error:
KeyError: 'content-type'
I checked the content of r.headers and it's:
CaseInsensitiveDict({'date': 'Fri, 20 May 2016 06:44:19 GMT', 'content-length': '0', 'connection': 'keep-alive', 'server': 'BigIP'})
What could be causing this?
Not all servers set a Content-Type header. Use .get() to retrieve a default if it is missing:
if "text/html" in r.headers.get("content-type", ''):
For the URL you gave I can't reproduce this:
$ curl -s -D - -o /dev/null "http://health.usnews.com/health-news/articles/2014/10/15/limiting-malpractice-claims-may-not-curb-costly-medical-tests"
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Powered-By: Brightspot
Content-Type: text/html;charset=UTF-8
Date: Fri, 20 May 2016 06:45:12 GMT
Set-Cookie: JSESSIONID=A0C35776067AABCF9E029150C64D8D91; Path=/; HttpOnly
Transfer-Encoding: chunked
but if the header is missing from your response then it usually isn't Python's fault, and certainly not your code's fault.
It could be you encountered a buggy server or temporary glitch, or the server you contacted doesn't like you for one reason or another. Your sample response headers have the content-length set to 0 as well, for example, indicating there was no content to serve at all.
The server that gave you that response is BigIP, a load balancer / network router product from a company called F5. Hard to say exactly what kind (they have global routing servers as well as per-datacenter or cluster load balancers). It could be that the load balancer ran out of back-end servers to serve the request, doesn't have servers in your region, or the load balancer decided that you are sending too many requests and refuses to give you more than just this response, or it is the wrong phase of the moon and Jupiter is in retrograde and it threw a tantrum. We can't know!
But, just in case this happens again, do also look at the response status code. It may well be a 4xx or 5xx status code indicating that something was wrong with your request or with the server. For example, a 429 status code response would indicate you made too many requests in a short amount of time and should slow down. Test for it by checking r.status_code.
Hello all… I am using Python to try to simulate login my work email, web base.
Now a problem lies on the first page of login. I observe the login process through IE9’s F12 (Developer tool) and found there are in total 3 cookies generated.
However when I use below scripts to visit the first page of login, there are only 2 cookies found. How could this be? How can I have all the cookies retrieved?
(I’ve cleared all the cache, cookie history, and empty the fold of Temporary Internet Files before catching)
loginUrl = "http://www.company.com/"
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
resp = urllib2.urlopen(loginUrl)
for index, cookie in enumerate(cj):
print '[',index, ']',cookie
Thanks.
This is because the first two cokies was returned from server read about this:
HTTP/1.1 200 OK
Content-type: text/html
Set-Cookie: foo=10
Set-Cookie: bar=20; Expires=Fri, 30 Sep 2011 11:48:00 GMT
the third one was generated by javascript:
$( document ).ready(function() {
});
I am testing my web-tornado application using Firefox's HttpRequest add-on but after I log in and receive my secure cookie data, I am not able to re-use it to consume protected methods.
This is my response data:
POST http://mylocalurl:8888/user/login
Content-Type: application/x-www-form-urlencoded
Login=mylogin;Pass=123
-- response -- 200 OK Content-Length: 33
Content-Type: text/html; charset=UTF-8
Server: TornadoServer/2.2.1
Set-Cookie:
IdUser="Mjk=|1395170421|ffaf0d6fecf2f91c0dccca7cab03d799ef6637a0";
expires=Thu, 17 Apr 2014 19:20:21 GMT; Path=/
{
"Success": true }
-- end response --
Now why I am trying to do is to configure HttpRequester to use this cookie for my new requests. I tried to add it using the "Headers" tab but my server keeps sending me a 403, Forbidden.
Can anyone help me on this ? It could be with another tool (for linux) too.
I really like fiddler2 for these kind of things and there's an alpha build for mono that you may wish to try out: http://www.telerik.com/download/fiddler
If you don't mind paid software you can use Charles, for which there is a free trial.
And if you are testing and already using python, why not use a simple python script with requests and its Session object with cookie-persistence..