Develop Reference

Can not get Set Cookie value python requests headers

I am trying to get a particular cookie from a request that I can see being set in browser through a particular endpoint which is like this
https://www.store.com/cart/miniCart/TOTAL?_=1591997339780
these are the response headers I see through Chrome Dev Tools
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
CF-Cache-Status: DYNAMIC
CF-RAY: 5a26c442fb22801a-SAN
cf-request-id: 034c18fddd0000801ac6b42200000001
Connection: keep-alive
Content-Encoding: gzip
Content-Language: es
Content-Type: application/json;charset=UTF-8
Date: Fri, 12 Jun 2020 21:46:48 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: 0
Pragma: no-cache
Server: cloudflare
Set-Cookie: JSESSIONID=18A8A12169ED6472A7359160F663CCF8; Path=/; Secure; HttpOnly
set-cookie: store-cart=d49a003e-41b5-444a-a71d-26b6f8db201c; Expires=Sun, 09-Nov-2031 13:46:48 GMT; Path=/; Secure; HttpOnly
set-cookie: AWSELB=11E5B3D30C8ACAF6D3240C8807474BBC740A29E2E0C61131788A04E3E6A646357EAA774C0A57B3DA33B571BADB93658470F13A3C847B4477CA237BB286CE5F3813ACBA53EEB69427F5D135043AFB3B2DC4835F3057;PATH=/;SECURE;HTTPONLY
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
Transfer-Encoding: chunked
Via: 1.1 www.innvictus.com
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
but through Python requests I get the following headers only with response.headers using exact request headers in my browser and in my code
{'Date': 'Fri, 12 Jun 2020 21:56:36 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Content-Language': 'es', 'Expires': '0', 'Pragma': 'no-cache', 'Set-Cookie': 'JSESSIONID=66AA2037611590192D2E13C38FF65289; Path=/; Secure; HttpOnly, AWSELB=11E5B3D30C8ACAF6D3240C8807474BBC740A29E2E0D0EAFB9AD200F275E3F63597988B98E611188683EDE09A5FA437554B92ECADED7B4477CA237BB286CE5F3813ACBA53EEF44544C6AD7FBBF8C242FCAC378603C5;PATH=/;SECURE;HTTPONLY', 'Strict-Transport-Security': 'max-age=31536000 ; includeSubDomains', 'Via': '1.1 www.store.com', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'CF-Cache-Status': 'DYNAMIC', 'cf-request-id': '034c21f9160000e6f09b961200000001', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Server': 'cloudflare', 'CF-RAY': '5a26d2a1beb9e6f0-EWR', 'Content-Encoding': 'gzip'}
the cookie I need is the "store-cart=d49a003e-41b5-444a-a71d-26b6f8db201c; Expires=Sun, 09-Nov-2031 13:46:48" cookie but as you can see not in the dictionary I get with response.headers

Different JSON repsonse when using requests module in python

I am trying to get JSON response from this URL.
But the JSON I see in the browser is different than what I get from python's requests response.
The code and its output:-
#code
import requests
r = requests.get("https://www.bigbasket.com/product/get-products/?slug=fruits-vegetables&page=1&tab_type=[%22all%22]&sorted_on=popularity&listtype=pc")
print("Status code: ", r.status_code)
print("JSON: ", r.json())
print("Headers:\n",r.headers())
#output
Status code: 200
JSON: '{"cart_info": {}, "tab_info": [], "screen_name": ""}'
Headers:
{'Content-Type': 'application/json',
'Content-Length': '52',
'Server': 'nginx',
'x-xss-protection': '1; mode=block',
'x-content-type-options': 'nosniff',
'x-frame-options': 'SAMEORIGIN',
'Access-Control-Allow-Origin': 'https://b2b.bigbasket.com',
'Date': 'Sat, 02 Sep 2017 18:43:51 GMT',
'Connection': 'keep-alive',
'Set-Cookie': '_bb_cid=4; Domain=.bigbasket.com; expires=Fri, 28-Aug-2037 18:43:51 GMT; Max-Age=630720000; Path=/, ts="2017-09-03 00:13:51.164"; Domain=.bigbasket.com; expires=Sun, 02-Sep-2018 18:43:51 GMT; Max-Age=31536000; Path=/, _bb_rd=6; Domain=.bigbasket.com; expires=Sun, 02-Sep-2018 18:43:51 GMT; Max-Age=31536000; Path=/'}
This is what Chrome shows in dev tools:-
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 4206
Server: nginx
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
Content-Encoding: gzip
x-frame-options: SAMEORIGIN
Access-Control-Allow-Origin: https://b2b.bigbasket.com
Date: Sat, 02 Sep 2017 15:43:20 GMT
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: ts="2017-09-02 21:13:20.193"; Domain=.bigbasket.com; expires=Sun, 02-Sep-2018 15:43:20 GMT; Max-Age=31536000; Path=/
Set-Cookie: _bb_rd=6; Domain=.bigbasket.com; expires=Sun, 02-Sep-2018 15:43:20 GMT; Max-Age=31536000; Path=/
Also tried separating query string and specifying it as params argument but it is giving the same result.

import requests
s = requests.session()
s.get("https://www.bigbasket.com/product/get-products/?slug=fruits-vegetables&page=1&tab_type=[%22all%22]&sorted_on=popularity&listtype=pc")
r = s.get("https://www.bigbasket.com/product/get-products/?slug=fruits-vegetables&page=1&tab_type=[%22all%22]&sorted_on=popularity&listtype=pc")
print("Status code: ", r.status_code)
print("JSON: ", r.json())

This is happening because of different City ID identified by your web browser and Requests.
You can check value of _bb_cid in both the cases

403 error when accessing API from server and not from browser

I am trying to access the Buxfer REST API using Python and urllib2.
The issue is I get the following response:
urllib2.HTTPError: HTTP Error 403: Forbidden
But when I try the same call through my browser, it works fine...
The script goes as follows:
username = "xxx#xxxcom"
password = "xxx"
#############
def checkError(response):
result = simplejson.load(response)
response = result['response']
if response['status'] != "OK":
print "An error occured: %s" % response['status'].replace('ERROR: ', '')
sys.exit(1)
return response
base = "https://www.buxfer.com/api";
url = base + "/login?userid=" + username + "&password=" + password;
req = urllib2.Request(url=url)
response = checkError(urllib2.urlopen(req))
token = response['token']
url = base + "/budgets?token=" + token;
req = urllib2.Request(url=url)
response = checkError(urllib2.urlopen(req))
for budget in response['budgets']:
print "%12s %8s %10.2f %10.2f" % (budget['name'], budget['currentPeriod'], budget['limit'], budget['remaining'])
sys.exit(0)
I also tried using the requests library but the same error appears.
The server I am tring to access from is an ubuntu 14.04, any help on explaining or solving why this happens will be appreciated.
EDIT:
This is the full error message:
{
'cookies': <<class 'requests.cookies.RequestsCookieJar'>[]>,
'_content': '
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /api/login
on this server.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at www.buxfer.com Port 443</address>
</body></html>',
headers': CaseInsensitiveDict({
'date': 'Sun, 31 Jan 2016 12:06:44 GMT',
'content-length': '291',
'content-type': 'text/html; charset=iso-8859-1',
'server': 'Apache/2.4.7 (Ubuntu)'
}),
'url': u'https://www.buxfer.com/api/login?password=xxxx&userid=xxxx%40xxxx.com',
'status_code': 403,
'_content_consumed': True,
'encoding': 'iso-8859-1',
'request': <PreparedRequest [GET]>,
'connection': <requests.adapters.HTTPAdapter object at 0x7fc7308102d0>,
'elapsed': datetime.timedelta(0, 0, 400442),
'raw': <urllib3.response.HTTPResponse object at 0x7fc7304d14d0>,
'reason': 'Forbidden',
'history': []
}
EDIT 2: (Network parameters in GoogleChrome browser)
Request Method:GET
Status Code:200 OK
Remote Address:52.20.61.39:443
Response Headers
view parsed
HTTP/1.1 200 OK
Date: Mon, 01 Feb 2016 11:01:10 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Cache-Controle: no-cache
Set-Cookie: login=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=buxfer.com
Set-Cookie: remember=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=buxfer.com
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/x-javascript; charset=utf-8
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8,fr;q=0.6
Connection:keep-alive
Cookie:PHPSESSID=pjvg8la01ic64tkkfu1qmecv20; api-session=vbnbmp3sb99lqqea4q4iusd4v3; __utma=206445312.1301084386.1454066594.1454241953.1454254906.4; __utmc=206445312; __utmz=206445312.1454066594.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
Host:www.buxfer.com
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36
EDIT 3:
I can also access through my local pycharm console without any issues, it's just when I try to do it from my remote server...

It could be that you need to do a POST rather than a GET request. Most logins work this way.
Using the requests library, you would need
response = requests.post(
base + '/login',
data={
'userid': username,
'password': password
}
)

To affirm #Carl from the offical website:
This command only looks at POST parameters and discards GET parameters.
https://www.buxfer.com/help/api#login

Form Submission/POST using Requests in Python

I'm trying to submit a form to a website using the requests module in Python but my form is not submitting correctly. I can submit the form correctly manually on the site but I assume something is wrong with my code that is causing the Python submission to fail. I can successfully login to the website and visit pages/issue GET requests using Python. I can issue a GET request to the page that I submit the form to and it will successfully load, i.e. the requests login works. I have included all of my output below including the Python code, an invalid form submission in Python and a valid form submission from the browser. This may be overkill but I am inexperienced with this and am not sure what is necessary.
My code to login is:
s = requests.Session()
data = s.get(login_url)
authToken = re.search(('name="authenticity_token"[\s]'
'type="hidden"[\s]+value="(.+)"'), \
data.text).group(1)
data_dict = {
'utf8': '✓',
'authenticity_token': authToken,
'admin[email]': username,
'admin[password]': password,
'admin[remember_me]': '1',
'commit': 'Sign in'
}
s.post(login_url, data_dict)
This successfully logs me in and I can submit GET requests to any page and get valid results.
My code to submit the form:
payload = {
'utf8': '✓',
'authenticity_token': authToken,
'progress_course[name]': name,
'progress_course[description]': desc,
'commit': 'Create Course'
}
headers = {
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate',
'Accept-Language':'en-US,en;q=0.8',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Content-Length':'176',
'Content-Type':'application/x-www-form-urlencoded',
'Host':'xxx.com',
'Origin':'https://xxx.com',
'Referer':'https://xxx.com/workshop/progress/courses/new',
'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36'
}
response = s.post(path,data=payload,headers=headers)
My form submission does not work. Here is the python logging module output:
The POST:
send: 'POST /workshop/progress/courses HTTP/1.1
Origin: https://xxx.com\r\nContent-Length: 181
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Host: xxx.com
Referer: https://xxx.com/workshop/progress/courses/new
Cache-Control: max-age=0
Cookie: _brainfit_session=xxx; remember_admin_token=xxx
Content-Type: application/x-www-form-urlencoded
progress_course%5Bdescription%5D=pytest&utf8=%26%23x2713%3B&commit=Create+Course&progress_course%5Bname%5D=pytest&authenticity_token=xxx
The reply to the POST. You can see that this comes from the sign-in page rather than generating a new page as seen in the correct output further down.
reply: 'HTTP/1.1 302 Found\r\n'
header: Server: nginx
header: Date: Tue, 02 Dec 2014 01:57:46 GMT
header: Content-Type: text/html; charset=utf-8
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Status: 302 Found
header: Strict-Transport-Security: max-age=31536000
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Location: https://xxx.com/admins/sign_in
header: Cache-Control: no-cache
header: X-Request-Id: 983472d4-8954-4106-904a-38ea3b6a76a1
header: X-Runtime: 0.039963
DEBUG:requests.packages.urllib3.connectionpool:"POST /workshop/progress/courses HTTP/1.1" 302 None
Redirect GETs and replies. I may be mistaken in what these actually are:
send: 'GET /admins/sign_in HTTP/1.1
Origin: https://xxx.com
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate
Host: xxx.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Connection: keep-alive
Referer: https://xxx.com/workshop/progress/courses/new
Cache-Control: max-age=0
Cookie: _brainfit_session=xxx; remember_admin_token=xxx
Content-Type: application/x-www-form-urlencoded
reply: 'HTTP/1.1 302 Found\r\n'
header: Server: nginx
header: Date: Tue, 02 Dec 2014 01:57:46 GMT
header: Content-Type: text/html; charset=utf-8
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Status: 302 Found
header: Strict-Transport-Security: max-age=31536000
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Location: https://xxx.com/workshop/progress/courses
header: Cache-Control: no-cache
header: Set-Cookie: _brainfit_session=xxx; path=/; secure; HttpOnly
header: X-Request-Id: 9e5535d6-143c-4e98-8bb4-35dd29ab045d
header: X-Runtime: 0.007505
DEBUG:requests.packages.urllib3.connectionpool:"GET /admins/sign_in HTTP/1.1" 302 None
send: 'GET /workshop/progress/courses HTTP/1.1
Origin: https://xxx.com
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate
Host: xxx.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Connection: keep-alive
Referer: https://xxx.com/workshop/progress/courses/new
Cache-Control: max-age=0
Cookie: _brainfit_session=exxx; remember_admin_token=xxx
Content-Type: application/x-www-form-urlencoded
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 02 Dec 2014 01:57:46 GMT
header: Content-Type: text/html; charset=utf-8
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Status: 200 OK
header: Strict-Transport-Security: max-age=31536000
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Cache-Control: max-age=0, private, must-revalidate
header: Set-Cookie: _brainfit_session=xxx; path=/; secure; HttpOnly
header: X-Request-Id: 4e759b60-af1c-4f39-b033-71ff99b62df4
header: X-Runtime: 0.019001
header: Content-Encoding: gzip
DEBUG:requests.packages.urllib3.connectionpool:"GET /workshop/progress/courses HTTP/1.1" 200 None
The POST headers of a properly submitted form on the site:
Remote Address:xxx
Request URL:https://xxx.com/workshop/progress/courses
Request Method:POST
Status Code:302 Found
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:176
Content-Type:application/x-www-form-urlencoded
Cookie:__utma=xxx; __utmc=xxx; __utmz=xxx.utmcsr=google|utmccn=(organic)|utmcmd=organic|
utmctr=xxx; km_lv=x; kvcd=xxx; km_ai=xxx; km_ni=xxx; km_uq=xxx; has_logged_in=true; WT_FPC=id=xxx; _ga=xxx; _brainfit_session=xxx
Host:xxx.com
Origin:https://xxx.com
Referer:https://xxx.com/workshop/progress/courses
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Form Data
utf8:✓
authenticity_token:xxx
progress_course[name]:test03
progress_course[description]:test03
commit:Create Course
Response Headers
Cache-Control:no-cache
Connection:keep-alive
Content-Type:text/html; charset=utf-8
Date:Mon, 01 Dec 2014 14:01:37 GMT
Location:https://xxx.com/workshop/progress/courses/44
Server:nginx
Set-Cookie:_brainfit_session=xxx; path=/; secure; HttpOnly
Status:302 Found
Strict-Transport-Security:max-age=31536000
Transfer-Encoding:chunked
X-Content-Type-Options:nosniff
X-Frame-Options:SAMEORIGIN
X-Request-Id:8bb04242-bc5a-408f-99a5-9d8bf4eeb611
X-Runtime:0.028559
X-XSS-Protection:1; mode=block
Once the form is correctly submitted there is also a GET response from the page it redirects to. I've excluded additional output here but it can be added if it will help resolve the problem.
Remote Address:xxx
Request URL:https://xxx.com/workshop/progress/courses/44
Request Method:GET
Status Code:200 OK

'Missing' cookies in python requests response

I have a working cURL command, which gets a number of cookies from this particular (alfresco, although I don't think this is particularly an alfresco issue), however when I try and encode the same request with python requests, I get only one cookie.
The cURL request is
curl -v --junk-session-cookies -H 'Content-Type: application/x-www-form-urlencoded' --cookie-jar cookies.txt --cookie cookies.txt -H 'Origin: http://<url>' -D headers.txt -e ';auto' -X POST -d #credentials http://<url>/share/page/dologin
And the python is
r=s.post('%s/share/page/dologin' % <url>,
data=credentials, allow_redirects=False)
where
s.headers
Out[121]: {'Origin': <url>, 'Host' :<url>, 'referer':<url> , 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.1 CPython/2.7.5+ Linux/3.11.0-26-generic','Content-Type': 'application/x-www-form-urlencoded'}
designed to closely match the headers that cURL is sending in.
The allow_redirects=False is there to mirror the lack of the -L option.
cURL yields:
Added cookie JSESSIONID="<removed>" for domain 10.12.3.166, path /share/, expire 0
< Set-Cookie: JSESSIONID=<removed>; Path=/share/; HttpOnly
* Added cookie alfLogin="<removed>" for domain 10.12.3.166, path /share, expire 1413289198
< Set-Cookie: alfLogin=<removed>; Expires=Tue, 14-Oct-2014 12:19:58 GMT; Path=/share
* Added cookie alfUsername3="<obf>" for domain 10.12.3.166, path /share, expire 1413289198
< Set-Cookie: alfUsername3=<obf>; Expires=Tue, 14-Oct-2014 12:19:58 GMT; Path=/share
Whereas I only get the JSESSIONID cookie in requests.
Update
I foolishly failed to include the headers sent in the cURL command, and those received.
Those sent were
> User-Agent: curl/7.32.0
> Host: <url>
> Accept: */*
> Referer:
> Content-Type: application/x-www-form-urlencoded
> Origin: <url>
> Content-Length: 44
And the headers sent back are
HTTP/1.1 302 Found
Server: Apache-Coyote/1.1
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID=<rem>; Path=/share/; HttpOnly
Set-Cookie: alfLogin=<rem>; Expires=Tue, 14-Oct-2014 12:19:58 GMT; Path=/share
Set-Cookie: alfUsername3=<rem>; Expires=Tue, 14-Oct-2014 12:19:58 GMT; Path=/share
Location: <url>/share
The headers sent back to requests are
{'transfer-encoding': 'chunked', 'set-cookie': 'JSESSIONID=<rem>; Path=/share/; HttpOnly', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Wed, 08 Oct 2014 08:54:53 GMT', 'content-type': 'text/html;charset=ISO-8859-1'}

The solution was to remove the "Host" header and set the referer field to be ''. I then got all the cookies I needed, so thanks to all the helpful comments.