I'm using the following code to interact with a website:
session = requests.session()
get = session.get(LANDING_URL, headers=HEADERS)
post = session.post(LANDING_URL, headers=HEADERS, data=PARAMS)
I'm using the session object to preserve cookies between call, but the post request following the get request doesn't seem to use the session cookie. The below output is from pdb:
(Pdb) get.cookies
<RequestsCookieJar[Cookie(version=0, name='ASP.NET_SessionId', value=...)]>
(Pdb) post.cookies
<RequestsCookieJar[]>
(Pdb) session.cookies
<RequestsCookieJar[Cookie(version=0, name='ASP.NET_SessionId', value=...)]>
Does this mean that the post request isn't using the session cookie? If so, why not?
Page may use JavaScript to add cookies and requests can't run JavaScript.
Using get.cookies, post.cookies you display only cookies send from server, not send to server.
session should keep all cookies from previous requests and send them in POST request.
You can use httpbin.org. If you send GET request to httpbin.org/get or POST to httpbin.org/post then it sends you back (as JSON) all your headers, data, cookies, etc. There are other useful functions on httpbin.org
You can also install local proxy server like Charles or Man-In-The-Middle-Py and send request through proxy. You will see body and headers in proxy. You can use proxy with web browser and with your script to compare your requests with request from browser.
You can also check post.request.body, post.request.headers. I never used it but it should have body and headers sent to server.
Related
I have a site with multiple cities. The task is to parse the price from the certain city. How do I get cookies from browser and how to pass it into requests.get(url=url, headers=headers, cookies=cookies?
Usually, cookies are sent to the server via the cookie header, which is part of the HTTP request.
Here is an example of some python code that sends a cookie to the server.
import requests
cookies = {'key': 'value'}
r = requests.get("https://example.com", cookies=cookies)
I have a question about cookies from a website.
If I do a request manually on the Website, it hands over the header with cookies.
cookie is for example this:
cookie: laravel_session=eyJpdiI6IjBCMnNTdFNmV3lCawererwadsf0SXc9PSIsInZhbHVlIjoiTWtZUDduSkFXdzRobERtQ0YxdkxreEkzYzJtWFwvUEVyMHgyT2ljN2EyeXg3XasdfweUUwQzhGRWowVEp5VygsdgfewTQ2MGJmNjEzZmYyNmU4YjdhYTQyMzkyNTYzYjFjMTQ4MDQ5NjlkNDQ0ZGQ4NGRiZDE5YjNhYTRhMzA5ODMifQ%3D%3D
but how can I get that cookie with my request? I want to send a post request, with the data in it, but I dont know how to send this cookie with the request Header or how to generate it?I dont want to put it in manually it should get it from the website. or does it automatically send this cookie with the header?
Thank you!
Elias
I dont get the thing with the laravel_session cookie- where can i get it from?
This should help you with your question because this how send cookies with a python request. Even if this is from wikipedia it is an example to send cookies with a request.
import requests
cookies = {'enwiki_session': '17ab96bd8ffbe8ca58a78657a918558'}
r = requests.post('http://wikipedia.org', cookies=cookies)
If I understand your question correctly, you want to get a cookie from the current session, then send it along with a request. This should do that.
import requests
req_url = 'http://foo.bar.com'
req_data = {'some': 'data'}
session = requests.Session()
response = requests.get(req_url)
cookie_to_send = session.cookies.getdict()['cookie_name']
requests.post(url=req_url, data=req_data, cookie={'cookie_name': cookie_to_send})
I have a web bot that is trying to get a cookie.
The flow goes:
I go get captcha, and a csrftoken (cookie)
I solve captcha and send solution to server.
They send back the session id.
The session id is a response cookie; although I seem to not get it in python.
The POST request to the server looks like this:
cookies={'csrftoken': 'h1239phtluwrane',}
headers = {'foo': 'bar'}
session=requests.Session()
r=session.post(URL, headers=headers, data=data, cookies=cookies)
try:
cookies['sessionid']=session.cookies['sessionid']
except KeyError:
print("Error getting correct cookie. %s" %session.cookies)
Then in session.cookies there is only the csrftoken as a request cookie.... But no response cookie to be found.
On another note. This same exact code used to work but suddenly stopped working even though I did not edit it. I verified that the server methods did not change.
To get your response cookies do:
print(r.json()['cookies'])
#{'tasty_cookie': 'yum'}
I'm coding a little snippet to fetch data from a web page, and I'm currently behind a HTTP/HTTPS proxy. The requests are created like this:
headers = {'Proxy-Connection': 'Keep-Alive',
'Connection':None,
'User-Agent':'curl/1.2.3',
}
r = requests.get("https://www.google.es", headers=headers, proxies=proxyDict)
At first, neither HTTP nor HTTPS worked, and the proxy returned 403 after the request. It was also weird that I could do HTTP/HTTPS requests with curl, fetching packages with apt-get or browsing the web. Having a look at Wireshark I noticed some differences between the curl request and the Requests one. After setting User-Agent to a fake curl version, the proxy instantly lets me do HTTP requests, so I supposed the proxy filter requests by User-Agent.
So, now I know why my code fails, and I can do HTTP requests, but the code keep on failing with HTTPS. I set the headers the same way as with HTTP, but after looking at Wireshark, no headers are sent in the CONNECT message, so the proxy sees no User-Agent and returns an ACCESS DENIED response.
I think that if only I could send the headers with the CONNECT message, I could do HTTPS requests easily, but I'm breaking my head around how to tell Requests that I want to send that headers.
Ok, so I found a way after looking at http.client. It's a bit lower level than using Requests but at least it work.
def HTTPSProxyRequest(method, host, url, proxy, header=None, proxy_headers=None, port=443):
https = http.client.HTTPSConnection(proxy[0], proxy[1])
https.set_tunnel(host, port, headers=proxy_headers)
https.connect()
https.request(method, url, headers=header)
response = https.getresponse()
return response.read(), response.status
# calling the function
HTTPSProxyRequest('GET','google.com', '/index.html', ('myproxy.com',8080))
I'm trying to send a post request to a website to get a json response. I can see the json response in Chrome Inspector when I click on a link, but I can get it using requests.
Firstly I tried to used requests Session to get the cookies first and use them in the post request, to no avail.
session = requests.Session()
session.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
response = session.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997')
print response.text
Secondly I used Selenium+PhantomJS to get the cookies and used them in requests, no results!
browser = webdriver.PhantomJS(PHANTOMJS_PATH)
browser.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
all_cookie = {}
for cookie in browser.get_cookies():
all_cookie[cookie['name']] = cookie['value']
rep = requests.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997', cookies=all_cookie)
It only works when I manually take the cookies from Chrome.
I can't see what's the problem!
session = requests.Session()
session.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
response = session.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997')
print(response.json)
Using the json attribute will fetch the JSON response. You can also use requests to make a persistent session, so the cookies are provided.
response.cookies #The cookies attribute