Develop Reference

How to get information from xml after login with requests python

I am trying to get some data from an xml of a website which requires to login first. I try to get python requests Session and I am able to login using get method (shows 200). After that I try to get access to the xml, but it gets me 401.
So far I know that the server checks with each call whether the client is sending a JSESSIONID cookie and whether the VALUE matches the current session.
I manage to get the corresponding cookies and tried to send it via post method but still get 401.
Maybe I think to complicated. I also do not need to achieve this with requests. I would be glad to just get the information from the xml.
import requests
login_url = 'https://www.example.com/login
USER_NAME = 'user'
PASSWORD = 'pass'
xml = 'https://www.example.com/channel_index?123&p=web'
with requests.Session() as s:
response = s.get(login_url,auth = (USER_NAME, PASSWORD))
print(response)
r = s.get(xml)
cookies = s.cookies.get_dict()
r = s.post(xml, cookies = cookies)
print(r)

Python Requests Not Recieving Cookie

I have a web bot that is trying to get a cookie.
The flow goes:
I go get captcha, and a csrftoken (cookie)
I solve captcha and send solution to server.
They send back the session id.
The session id is a response cookie; although I seem to not get it in python.
The POST request to the server looks like this:
cookies={'csrftoken': 'h1239phtluwrane',}
headers = {'foo': 'bar'}
session=requests.Session()
r=session.post(URL, headers=headers, data=data, cookies=cookies)
try:
cookies['sessionid']=session.cookies['sessionid']
except KeyError:
print("Error getting correct cookie. %s" %session.cookies)
Then in session.cookies there is only the csrftoken as a request cookie.... But no response cookie to be found.
On another note. This same exact code used to work but suddenly stopped working even though I did not edit it. I verified that the server methods did not change.

To get your response cookies do:
print(r.json()['cookies'])
#{'tasty_cookie': 'yum'}

Cookies must be enabled in your browser [Python Requests]

So I'm trying to log into my hotmail account via python and keep getting this response on the page when I make this request
r = requests.post('https://login.live.com', auth=('Email', 'Pass'),verify=False)
Cookies must be allowed
Your browser is currently set to block cookies. Your browser must allow cookies before you can use a Microsoft account.
Cookies are small text files stored on your computer that tell Microsoft sites and services when you're signed in. To learn how to allow cookies, see online help in your web browser.
I would also like to mention that I am trying to httpPOST to this webpage because I would rather handle the cookies in the response and access other pages of my microsoft profile (rather than just accessing my email via the smtp server)
Thanks!
Edit :
import requests
s = requests.Session()
r = s.get('https://login.live.com',verify=False)
r = s.post('https://login.live.com', auth=('user', 'pass'),verify=False)
print r.status_code
print r.text

Use requests.Session to persist a session (with cookies included):
import requests
s = requests.Session()
res = s.get('https://login.live.com')
cookies = dict(res.cookies)
res = s.post('https://login.live.com',
auth=('Email', 'Password'),
verify=False,
cookies=cookies)

Can't emulate browser behavior with requests

I'm trying to send a post request to a website to get a json response. I can see the json response in Chrome Inspector when I click on a link, but I can get it using requests.
Firstly I tried to used requests Session to get the cookies first and use them in the post request, to no avail.
session = requests.Session()
session.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
response = session.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997')
print response.text
Secondly I used Selenium+PhantomJS to get the cookies and used them in requests, no results!
browser = webdriver.PhantomJS(PHANTOMJS_PATH)
browser.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
all_cookie = {}
for cookie in browser.get_cookies():
all_cookie[cookie['name']] = cookie['value']
rep = requests.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997', cookies=all_cookie)
It only works when I manually take the cookies from Chrome.
I can't see what's the problem!

session = requests.Session()
session.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
response = session.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997')
print(response.json)
Using the json attribute will fetch the JSON response. You can also use requests to make a persistent session, so the cookies are provided.
response.cookies #The cookies attribute

Update Cookies in Session Using python-requests Module

I'm using python-requests module to handle oAuth request and response.
I want to set received access_token (response content as dict) in requests.session.cookies object.
How can I update existing cookies of session with received response from server?
[EDIT]
self.session = requests.session(auth=self.auth_params)
resp = self.session.post(url, data=data, headers=self.headers)
content = resp.content
I want to do something like:
requests.utils.dict_from_cookiejar(self.session.cookies).update(content)
Here, requests.utils.dict_from_cookiejar(self.session.cookies) returns dict with one session key. Now, I want to update received response content in self.session.cookies.

requests can do that for you, provided you tell it all the requests you make are part of the same session:
>>> import requests
>>> s = requests.session()
>>> s.get('https://www.google.com')
<Response [200]>
>>> s.cookies
<<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='NID'...
Subsequent requests made using s.get or s.post will re-use and update the cookies the server sent back to the client.
To add a Cookie on your own to a single request, you would simply add it via the cookies parameter.
>>> s.get('https://www.google.com', cookies = {'cookieKey':'cookieValue'})
Unless the server sends back a new value for the provided cookie, the session will not retain the provided cookie.

In order to provide a cookie yourself to the requests module you can use the cookies parameter for a single request and give it a cookie jar or dict like object containing the cookie(s).
>>> import requests
>>> requests.get('https://www.example.com', cookies {'cookieKey':'cookieValue'})
But if you want to retain the provided cookie without having to set the cookies parameter everytime, you can use a reqests session which you can also pass to other funtions so they can use and update the same cookies:
>>> session = requests.session()
>>> session.cookies.set('cookieKey', 'cookieName')
# In order to avoid cookie collisions
# and to only send cookies to the domain / path they belong to
# you have to provide these detail via additional parameters
>>> session.cookies.set('cookieKey', 'cookieName', path='/', domain='www.example.com')

This code worked for me. hope it can help to someone else.
I want to update session.cookies variable with received response values from post request.
so, same request value can be used in another post/get request.
here, what I did:
1) updated requests module to 1.0.3 version.
2) created 2 functions
session = requests.session()
def set_SC(cookie_val):
for k,v in cookie_dict.iteritems():
if not isinstance(v, str):
cookie_dict[k] = str(v)
requests.utils.add_dict_to_cookiejar(session.cookies,
cookie_val)
def get_SC():
return requests.utils.dict_from_cookiejar(session.cookies)
In another function:
setSC(response.content)