I have a web bot that is trying to get a cookie.
The flow goes:
I go get captcha, and a csrftoken (cookie)
I solve captcha and send solution to server.
They send back the session id.
The session id is a response cookie; although I seem to not get it in python.
The POST request to the server looks like this:
cookies={'csrftoken': 'h1239phtluwrane',}
headers = {'foo': 'bar'}
session=requests.Session()
r=session.post(URL, headers=headers, data=data, cookies=cookies)
try:
cookies['sessionid']=session.cookies['sessionid']
except KeyError:
print("Error getting correct cookie. %s" %session.cookies)
Then in session.cookies there is only the csrftoken as a request cookie.... But no response cookie to be found.
On another note. This same exact code used to work but suddenly stopped working even though I did not edit it. I verified that the server methods did not change.
To get your response cookies do:
print(r.json()['cookies'])
#{'tasty_cookie': 'yum'}
Related
I have a question about cookies from a website.
If I do a request manually on the Website, it hands over the header with cookies.
cookie is for example this:
cookie: laravel_session=eyJpdiI6IjBCMnNTdFNmV3lCawererwadsf0SXc9PSIsInZhbHVlIjoiTWtZUDduSkFXdzRobERtQ0YxdkxreEkzYzJtWFwvUEVyMHgyT2ljN2EyeXg3XasdfweUUwQzhGRWowVEp5VygsdgfewTQ2MGJmNjEzZmYyNmU4YjdhYTQyMzkyNTYzYjFjMTQ4MDQ5NjlkNDQ0ZGQ4NGRiZDE5YjNhYTRhMzA5ODMifQ%3D%3D
but how can I get that cookie with my request? I want to send a post request, with the data in it, but I dont know how to send this cookie with the request Header or how to generate it?I dont want to put it in manually it should get it from the website. or does it automatically send this cookie with the header?
Thank you!
Elias
I dont get the thing with the laravel_session cookie- where can i get it from?
This should help you with your question because this how send cookies with a python request. Even if this is from wikipedia it is an example to send cookies with a request.
import requests
cookies = {'enwiki_session': '17ab96bd8ffbe8ca58a78657a918558'}
r = requests.post('http://wikipedia.org', cookies=cookies)
If I understand your question correctly, you want to get a cookie from the current session, then send it along with a request. This should do that.
import requests
req_url = 'http://foo.bar.com'
req_data = {'some': 'data'}
session = requests.Session()
response = requests.get(req_url)
cookie_to_send = session.cookies.getdict()['cookie_name']
requests.post(url=req_url, data=req_data, cookie={'cookie_name': cookie_to_send})
I'm using the following code to interact with a website:
session = requests.session()
get = session.get(LANDING_URL, headers=HEADERS)
post = session.post(LANDING_URL, headers=HEADERS, data=PARAMS)
I'm using the session object to preserve cookies between call, but the post request following the get request doesn't seem to use the session cookie. The below output is from pdb:
(Pdb) get.cookies
<RequestsCookieJar[Cookie(version=0, name='ASP.NET_SessionId', value=...)]>
(Pdb) post.cookies
<RequestsCookieJar[]>
(Pdb) session.cookies
<RequestsCookieJar[Cookie(version=0, name='ASP.NET_SessionId', value=...)]>
Does this mean that the post request isn't using the session cookie? If so, why not?
Page may use JavaScript to add cookies and requests can't run JavaScript.
Using get.cookies, post.cookies you display only cookies send from server, not send to server.
session should keep all cookies from previous requests and send them in POST request.
You can use httpbin.org. If you send GET request to httpbin.org/get or POST to httpbin.org/post then it sends you back (as JSON) all your headers, data, cookies, etc. There are other useful functions on httpbin.org
You can also install local proxy server like Charles or Man-In-The-Middle-Py and send request through proxy. You will see body and headers in proxy. You can use proxy with web browser and with your script to compare your requests with request from browser.
You can also check post.request.body, post.request.headers. I never used it but it should have body and headers sent to server.
I am attempting to get user statistics from the Fortnite tracker api.
I have an api key and am using the correct url as indicated in the documentation
Template url:
https://api.fortnitetracker.com/v1/profile/{platform}/{epic-nickname}
Desired url:
https://api.fortnitetracker.com/v1/profile/pc/xantium0
If I use this link in browser I get {"message":"No API key found in request"} (as I have not passed the API key) so the link should be correct. Also if I do not pass the api key with urllib then I still get a 403 error.
I have checked out how to pass a header in a request: How do I set headers using python's urllib?
and so far have this code:
import urllib.request as ur
request = ur.Request('https://api.fortnitetracker.com/v1/profile/pc/xantium0', headers={'TRN-Api-Key' : 'xxx'})
response = ur.urlopen(request)
print(response.read())
When run I get this error:
urllib.error.HTTPError: HTTP Error 403: Forbidden
403 checks out as:
HTTP 403 is a standard HTTP status code communicated to clients by an HTTP server to indicate that the server understood the request, but will not fulfill it. There are a number of sub-status error codes that provide a more specific reason for responding with the 403 status code.
https://en.wikipedia.org/wiki/HTTP_403
The response is the same if I don't pass the api key in the header.
I can only think of three reasons this code is not working:
I have passed the wrong header name (i.e. it's not TRN-Api-Key)
My code is incorrect and I am not actually passing a header to the server
I have been banned
My problem is that I think my code is correct:
From the documentation:
urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)
I have passed the url and I have passed the headers (wihout confusing with the data arguement). The api documentation also mentions it should be passed in the headers.
I am also quite sure I need to use the TRN-Api-Key as it is shown in the api documentation:
TRN-Api-Key: xxx
Also in this question (using Ruby):
header = {
key: "TRN-Api-Key: Somelong-api-key-here"
}
Or I have been banned (this is possible although I got the key 15 minutes ago) is there a way to check? Would this error be returned?
What is preventing me from getting the user statistics?
Try using requests, a pythonic, fast and widely used module.
import requests
url = 'https://api.fortnitetracker.com/v1/profile/pc/xantium0'
headers = {
'TRN-Api-Key' : 'xxx'
}
response = requests(url, headers=headers)
print('Requests was successful:', response.ok)
print(response.text)
If it doesn't work you can visit the url with your browser, then check the requests:
in Firefox press Cntrl+Shift+E, in Chrome Cntrl+E (or Inspect with Cntrl+Shift+I and then go to Network). Press on "https://api.fortnitetracker.com/v1/profile/pc/xantium0" and change the headers. On Firefox there's the button Modify and resend. Check the response and eventually, try to change the header api key name.
Hope this helps, let me know.
I'm trying to send a post request to a website to get a json response. I can see the json response in Chrome Inspector when I click on a link, but I can get it using requests.
Firstly I tried to used requests Session to get the cookies first and use them in the post request, to no avail.
session = requests.Session()
session.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
response = session.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997')
print response.text
Secondly I used Selenium+PhantomJS to get the cookies and used them in requests, no results!
browser = webdriver.PhantomJS(PHANTOMJS_PATH)
browser.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
all_cookie = {}
for cookie in browser.get_cookies():
all_cookie[cookie['name']] = cookie['value']
rep = requests.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997', cookies=all_cookie)
It only works when I manually take the cookies from Chrome.
I can't see what's the problem!
session = requests.Session()
session.get('http://www.auchandrive.fr/drive/pagestatique.pagetemplate.popuphandler.popinchangementmagasin.changermag/537?t:ac=PAGE_STATIQUE_ENGAGEMENTS')
response = session.post('http://www.auchandrive.fr/drive/rayon.productlist.pagination_0.topage/1?t:ac=3686973/3686997')
print(response.json)
Using the json attribute will fetch the JSON response. You can also use requests to make a persistent session, so the cookies are provided.
response.cookies #The cookies attribute
I am using python requests library for a POST request and I expect a return message with an empty payload. I am interested in the headers of the returned message, specifically the 'Location' attribute. I tried the following code:
response=requests.request(method='POST', url=url, headers={'Content-Type':'application/json'}, data=data)
print response.headers ##Displays a case-insensitve map
print response.headers['Location'] ##blows up
Strangely the 'Location' attribute is missing in the headers map. If I try the same POST request on postman, I do get a valid Location attribute. Has anyone else seen this? Is this a bug in the requests library?
Sounds like everything's working as expected? Check your response.history
From the Requests documentation:
Requests will automatically perform location redirection for all verbs except HEAD.
>>> r = requests.get('http://github.com')
>>> r.url
'https://github.com/'
>>> r.status_code
200
>>> r.history
[<Response [301]>]
From the HTTP Location page on wikipedia:
The HTTP Location header field is returned in responses from an HTTP server under two circumstances:
To ask a web browser to load a different web page. In this circumstance, the Location header should be sent with an HTTP status code of 3xx. It is passed as part of the response by a web server when the requested URI has:
Moved temporarily, or
Moved permanently
To provide information about the location of a newly-created resource. In this circumstance, the Location header should be sent with an HTTP status code of 201 or 202.1
The requests library follows redirections automatically.
To take a look at the redirections, look at the history of the requests. More details in the docs.
Or you pass the extra allow_redirects=False parameter when making the request.