This is the code that is important:
elif msgheader.startswith( 'POST' ):
print "The POST msg:\n", msgheader
And this is the response:
The POST msg:
POST / HTTP/1.1
Host: 192.168.1.102:63166
Connection: keep-alive
Content-Length: 12
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://192.168.1.102:63166
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://192.168.1.102:63166/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
comment=test
Now my question is how i can print out individual rows of the response like the comment field. I tried using this code but i get an error:
print['comment']
Exception (with type 'exceptions.TypeError'): string indices must be integers, not str
You have to use something like this supposing you are using urllib2:
print msgheader.getheader('content-type')
With requests lib you should use:
print msgheader['content-type']
Related
On the instagram login page, if one inspects the element of the POST call for the url 'https://www.instagram.com/accounts/web_create_ajax/', it lists the following as headers:
Host: www.instagram.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.instagram.com/
X-CSRFToken: 7dmO9F3JuVGvSXumd79yByPxnHoWHz1A
X-Instagram-AJAX: c2d8f4380025
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Content-Length: 102
Cookie: csrftoken=7dmO9F3JuVGvSXumd79yByPxnHoWHz1A; mid=W30zsQAEAAErXHJ3iUojfTceCd53; mcd=3; csrftoken=7dmO9F3JuVGvSXumd79yByPxnHoWHz1A; rur=FTW
Connection: keep-alive
I am wondering if anyone would have any idea what X-Instagram-AJAX is and how I can generate it each time. Is it connected as a pair with X-CSRFToken? Thanks.
Follow, like etc requests working without this header. I don't know what is it but i think instagram dedects suspicious requests with this and then log it. You can get this value on any page in instagram This is x-instagram-ajax value
You can parse it and use.
I am beginner in python. I would like to parse a website but the header shows the content type text/x-gwt-rpc; charset=utf-8 and the request payload...
7|0|4|https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/|53263EDF7F9313FDD5BD38B49D3A7A77|com.liquable.hiroba.gwt.client.square.IGwtSquareService|getNoOfCrowd|1|2|3|4|0|
Request:
POST /com.liquable.hiroba.gwt.server.GWTHandler/squareService HTTP/1.1
Host: kekeke.cc
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: */*
Accept-Language: zh-TW,zh;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://kekeke.cc/
Content-Type: text/x-gwt-rpc; charset=utf-8
X-GWT-Permutation: 8F22796231EB8C8312C5D1BB10451262
X-GWT-Module-Base: https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/
Content-Length: 177
DNT: 1
Connection: keep-alive
Can anyone tell me how to make post request in python?
I found the solution. It can be solved by simply using r = requests.post(url, "7|0|4|https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/|53263EDF7F9313FDD5BD38B49D3A7A77|com.liquable.hiroba.gwt.client.square.IGwtSquareService|getNoOfCrowd|1|2|3|4|0|", headers=headers). In many online tutorials, they teach the post request by using data in form of {data:data} to submit post request only. However, it can be done by submitting data in form of string in some cases.
I've got problem while trying to post the file to the server. I'm trying to make file upload script to server, this server is very 'Sensitive to correctness post request'
I debugged page that is sending the file to server and browser send this (TextView):
POST http://example.com/post HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 20625
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarykGHBkXoER9gNuVna
Referer: http://example.com/foo
Accept-Encoding: gzip, deflate
Accept-Language: pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4,pt;q=0.2
------WebKitFormBoundarykGHBkXoER9gNuVna
Content-Disposition: form-data; name="files[]"; filename="file.zip"
Content-Type: application/octet-stream
...raw file data...
------WebKitFormBoundarykGHBkXoER9gNuVna--
However, my script is sending this (TextView):
POST http://example.com/post HTTP/1.1
Host: example.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.18.1
Content-Length: 20604
--f8c266cf436941019c5a80c7d4779a57
Content-Disposition: form-data; name="files[]"; filename="file.zip"
Content-Type: application/zip
...raw file data...
--f8c266cf436941019c5a80c7d4779a57--
With causes error on server, additional note: this error started when I changed files=files to data=files
Current Code:
files = MultipartEncoder({'files[]': (filename, open(local_path,'rb'), mimetype)})
UploadFile = requests.post(self.UploadURL, data=files, allow_redirects=False)
Working code:
files = {'files[]': (filename, open(local_path,'rb'), mimetype)}
UploadFile = requests.post(self.UploadURL, files=files, allow_redirects=False)
I'm using MultipartEncoder to allow sending huge files.
I see that biggest mismatch is "boundary", but why this 'boundary' is generating in working code but in Current code not?
How to fix that?
You are not setting the Content-Type header, the MultipartEncoder provides it for you:
files = MultipartEncoder({'files[]': (filename, open(local_path,'rb'), mimetype)})
UploadFile = requests.post(
self.UploadURL, data=files, allow_redirects=False,
headers={'Content-Type': files.content_type})
The header must come from the multi-part encoding, because it is responsible for picking the boundary used to deliniate the various MIME parts in the multipart response. In your upload that's:
--f8c266cf436941019c5a80c7d4779a57
but it is generated at random each time your code runs. The header provided would look like:
Content-Type: multipart/form-data; boundary=--f8c266cf436941019c5a80c7d4779a57
Using the RequestForm, I need to specify that the Content-Type is application/json; charset=UTF-8 and Accept is */*.
How to do this?
Currently, my code looks like this:
yield scrapy.FormRequest(url='...',
formdata={
...
},
cookies={...},
callback=self.parse_second)
Using browser, the request is:
POST /PaginasPublicas/_SBC.aspx/pesquisaLoteIntegracaoTPCL HTTP/1.1
Host: geosampa.prefeitura.sp.gov.br
Connection: keep-alive
Content-Length: 118
Accept: */*
Origin: http://geosampa.prefeitura.sp.gov.br
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
Content-Type: application/json; charset=UTF-8
Referer: http://geosampa.prefeitura.sp.gov.br/PaginasPublicas/_SBC.aspx
Accept-Encoding: gzip, deflate
Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4,ar;q=0.2,de;q=0.2,es;q=0.2,fr;q=0.2,it;q=0.2,ja;q=0.2,pl;q=0.2,tr;q=0.2,zh-TW;q=0.2
Cookie: ASP.NET_SessionId=bvvghxvsxgwzuyaudsqn5m5q
Your request should be like this:
yield FormRequest(..., headers={'Content-Type': 'application/json','charset':'UTF-8'})
Scrapy Request has a field headers which is use to define explicit headers. This will work for you.
yield scrapy.FormRequest(url='...',
formdata={
...
},
cookies={...}, headers={'Content-Type': 'application/json','charset':'UTF-8'},
callback=self.parse_second)
I'm trying to get single sign-on working in Azure Active Directory, using this bit of documentation as a guide. However, when I get up to the "access token request" stage, I get the following error:
Error validating credentials. AADSTS70000: The provided access grant is invalid or malformed.
Searching the Internet, it seems that this is usually caused by the redirect_uri parameter being missing in the second step or different between the two steps, but that doesn't seem to be the case here.
Here's what's happening in each step:
Authorize step (raw HTTP request from browser):
GET /[snip tenant id]/oauth2/authorize?redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Fpost-login%3Fdest%3D%252F&response_type=code&client_id=[snip client id] HTTP/1.1
Host: login.windows.net
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-AU,en;q=0.8,en-US;q=0.6
Cookie: [snip a handful of cookies]
Redirect step (raw HTTP request from browser):
GET /post-login?code=[snip base64]&session_state=[snip uuid] HTTP/1.1
Host: localhost:5000
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-AU,en;q=0.8,en-US;q=0.6
Cookie: csrftoken=vBjLMAFTw7NSFEJHb2t9GTA0Eoced4rw; azure-redirect-uri="http://localhost:5000/post-login?dest=%2F"
Token request step (raw HTTP request from server code):
POST /[snip tenant id]/oauth2/token HTTP/1.1
Host: login.windows.net
Content-Length: 805
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.5.0 CPython/2.7.6 Darwin/14.0.0
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Fpost-login%3Fdest%3D%252F&client_secret=Dsysz7F%2FXh2Wu1YKE%2BVEOkvMHhvc38DnwFTa5qekyXM%3D&code=[snip base64]&client_id=[snip client id]&grant_type=authorization_code
Response to token request (Python dict parsed from JSON returned):
{
u'timestamp': u'2014-12-09 05:37:58Z',
u'trace_id': u'ae00a782-30f1-4e1c-a183-f19330ecca37',
u'submit_url': None,
u'correlation_id': u'21a7b861-5171-4083-9da5-67e7d956ab5e',
u'error_description': u'AADSTS70002: Error validating credentials. AADSTS70000: The provided access grant is invalid or malformed.\r\nTrace ID: ae00a782-30f1-4e1c-a183-f19330ecca37\r\nCorrelation ID: 21a7b861-5171-4083-9da5-67e7d956ab5e\r\nTimestamp: 2014-12-09 05:37:58Z',
u'context': None,
u'error': u'invalid_grant',
u'error_codes': [70002, 70000]
}
Your token request is failing because of the query parameter at the end of your redirect uri. OAuth redirect URI's should not have any query parameters or fragments. You can use the state parameter instead.
I don't think that the authorize call should have succeeded either, but clearly it did. I am checking to see if that is a bug.
See this link for some more information on how to use the OAuth state parameter:
http://www.thread-safe.com/2014/05/the-correct-use-of-state-parameter-in.html