I want to make a program where I can send HTTP post requests and respond.
So, I want to send THIS post:
POST https: //example.com/index.php?s=&&app=box&module=ajax§ion=coreAjax&secure_key=&type=submit&lastid=87311&global=1 HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 10
Accept: text/javascript, text/html, application/xml, text/xml, */*
X-Prototype-Version: 1.7.2
Origin: https://example.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Content-type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://x.com/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: cookieconsent_status=dismiss;
And then enter the request body:
message= # Which I will make: "message= %s" % (messagex))
But I do not know how to send them and can´t seem to find any way online, could someone help please?
The main parts are:
import requests # you have to install this library, with pip for example
# define your custom headers (as many as you want)
headers = {
'X-Prototype-Version': '1.7.2'
}
# define your URL params (!= of the body of the POST request)
params = {
'your_first_param': 'its_value',
'your_second_param': 'its_value'
}
# define the body of the POST request
data = {
'message' : 'your message'
}
# send the POST request
response = requests.post('https://example.com/index.php', params=params, data=data, headers=headers)
# here is the response
print response.text
Hope that helps.
Related
I want to get the data points from the graph below (https://index.mysteel.com/price/getChartMultiCity_1_0.html)
I found out from Developer Tools -> Network -> XHR that a request is made when I explore the graph. The response has all the data needed, like date and price.
If I just copy the request URL and fetch the data with python's requests, I get the correct response. Example:
import requests
r = requests.get('https://index.mysteel.com/zs/newprice/getBaiduChartMultiCity.ms?catalog=%25E8%259E%25BA%25E7%25BA%25B9%25E9%2592%25A2_%3A_%25E8%259E%25BA%25E7%25BA%25B9%25E9%2592%25A2&city=%25E4%25B8%258A%25E6%25B5%25B7%3A15278&spec=HRB400E%252020MM_%3A_HRB400E_20MM&startTime=2021-08-10&endTime=2021-08-12&callback=json')
r.json()
{'marketDatas': .........}
However, if I change the values of endTime or startTime query parameters, I get a 401 error:
Missing authentication parameters!
It looks like I can only send requests that are already sent from my browser (I'm running a jupyter notebook in the same browser). Even if I add request headers like in the Network tab, I still get the same error:
params= {'catalog':'%E8%9E%BA%E7%BA%B9%E9%92%A2_:_%E8%9E%BA%E7%BA%B9%E9%92%A2' ,'city':'%E4%B8%8A%E6%B5%B7:15278' ,
'spec':'HRB400E%2020MM_:_HRB400E_20MM' ,'startTime':'2021-08-10', 'endTime':'2021-08-12', 'callback':'json'}
headers = {'Host': 'index.mysteel.com',
'User-Agent': 'Mozilla/5.0',
'Referer': 'https://index.mysteel.com/price/getChartMultiCity_1_0.html',
'appKey': '47EE3F12CF0C443F851EFDA73AC815',
'Cookie':'href=https%3A%2F%2Findex.mysteel.com%2Fprice%2FgetChartMultiCity_1_0.html; accessId=5d36a9e0-919c-11e9-903c-ab24d11b; pageViewNum=2'}
url = 'https://index.mysteel.com/zs/newprice/getBaiduChartMultiCity.ms'
r = requests.get(url, params=params, headers=headers)
What am I missing here? I am not authenticating myself in the browser, so there shouldn't be any authentication issues. Is there some other parameter missing in my headers dict?
PS: These are all the request headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,sq;q=0.8,de;q=0.7
appKey: 47EE3F12CF0C443F851EFDA73AC815
Connection: keep-alive
Cookie: href=https%3A%2F%2Findex.mysteel.com%2Fprice%2FgetChartMultiCity_1_0.html; accessId=5d36a9e0-919c-11e9-903c-ab24d11b; pageViewNum=2
dnt: 1
Host: index.mysteel.com
Referer: https://index.mysteel.com/price/getChartMultiCity_1_0.html
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
sec-gpc: 1
sign: D404184969BF3B8C081A9F0C913AF68E
timestamp: 1629136315665
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36
version: 1.0.0
X-Requested-With: XMLHttpRequest
I am beginner in python. I would like to parse a website but the header shows the content type text/x-gwt-rpc; charset=utf-8 and the request payload...
7|0|4|https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/|53263EDF7F9313FDD5BD38B49D3A7A77|com.liquable.hiroba.gwt.client.square.IGwtSquareService|getNoOfCrowd|1|2|3|4|0|
Request:
POST /com.liquable.hiroba.gwt.server.GWTHandler/squareService HTTP/1.1
Host: kekeke.cc
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: */*
Accept-Language: zh-TW,zh;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://kekeke.cc/
Content-Type: text/x-gwt-rpc; charset=utf-8
X-GWT-Permutation: 8F22796231EB8C8312C5D1BB10451262
X-GWT-Module-Base: https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/
Content-Length: 177
DNT: 1
Connection: keep-alive
Can anyone tell me how to make post request in python?
I found the solution. It can be solved by simply using r = requests.post(url, "7|0|4|https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/|53263EDF7F9313FDD5BD38B49D3A7A77|com.liquable.hiroba.gwt.client.square.IGwtSquareService|getNoOfCrowd|1|2|3|4|0|", headers=headers). In many online tutorials, they teach the post request by using data in form of {data:data} to submit post request only. However, it can be done by submitting data in form of string in some cases.
Here is what my XHR data looks like when captured in chrome
Request Header
POST my_url?X-Progress-ID=ee821652321919bc7ae61fbe0b625990&userpkgname=file_name.tar.gz HTTP/1.1
Host: 10.110.134.28
Connection: keep-alive
Content-Length: 17461
Accept: application/json, text/plain, */*
Origin: https://10.110.134.28
X-XSRF-TOKEN: bGIwdfFE-oaL_1yVrCzw0iHvv4yUHLC28xjw
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarybfK9jSdLoc2Mpj0i
Referer: https://10.110.134.28/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: XSRF-TOKEN=bGIwdfFE-oaL_1yVrCzw0iHvv4yUHLC28xjw; sid=s%3AXglfJNLQ9zzp3eHjQ2QOpk19kFKDDMvJ.ZMKyZd1Gx13lz2MnJgty5WncnilySzfoThGktkhlk4w
Payload
------WebKitFormBoundarybfK9jSdLoc2Mpj0i
Content-Disposition: form-data; name="package"; filename="file_name.tar.gz"
Content-Type: application/x-gzip
------WebKitFormBoundarybfK9jSdLoc2Mpj0i--
And this is how I am building my request.
files = {'package': (<file_name>, open(config_path, 'rb'), 'application/x-gzip')}
request.post(url, files=files)
This is how my request header looks like
{
'Content-Length' : '17449',
'Accept-Encoding' : 'gzip, deflate',
'Accept' : '*/*',
'User-Agent' : 'python-requests/2.10.0',
'Connection' : 'keep-alive',
'Cookie' : 'XSRF-TOKEN=zmklLEL0-gDJOfNBk113MuTpBkLo0j6MAzw0; sid=s%3AC2JZDCfpg_CgkU7qSlS5YTvWXwpgMX35.5nU7W02TPNYtMkIQ4W%2B1bjd87A7KyJbh3shoNqqADXE',
'Content-Type' : 'multipart/form-data; boundary=270d9e02bf214dc7a09c3081cba5b0e0',
'XSRF-TOKEN' : 'zmklLEL0-gDJOfNBk113MuTpBkLo0j6MAzw0'
}
When I make the request I get 502 bad gateway response that too after few seconds while on chrome I get 200 OK instantly
So most probably I am not building my request correctly. Any suggestions?
Using the RequestForm, I need to specify that the Content-Type is application/json; charset=UTF-8 and Accept is */*.
How to do this?
Currently, my code looks like this:
yield scrapy.FormRequest(url='...',
formdata={
...
},
cookies={...},
callback=self.parse_second)
Using browser, the request is:
POST /PaginasPublicas/_SBC.aspx/pesquisaLoteIntegracaoTPCL HTTP/1.1
Host: geosampa.prefeitura.sp.gov.br
Connection: keep-alive
Content-Length: 118
Accept: */*
Origin: http://geosampa.prefeitura.sp.gov.br
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
Content-Type: application/json; charset=UTF-8
Referer: http://geosampa.prefeitura.sp.gov.br/PaginasPublicas/_SBC.aspx
Accept-Encoding: gzip, deflate
Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4,ar;q=0.2,de;q=0.2,es;q=0.2,fr;q=0.2,it;q=0.2,ja;q=0.2,pl;q=0.2,tr;q=0.2,zh-TW;q=0.2
Cookie: ASP.NET_SessionId=bvvghxvsxgwzuyaudsqn5m5q
Your request should be like this:
yield FormRequest(..., headers={'Content-Type': 'application/json','charset':'UTF-8'})
Scrapy Request has a field headers which is use to define explicit headers. This will work for you.
yield scrapy.FormRequest(url='...',
formdata={
...
},
cookies={...}, headers={'Content-Type': 'application/json','charset':'UTF-8'},
callback=self.parse_second)
I think I'm on the right track for ASP.NET authentication. I'm trying to use requests to pass credentials to a website. Here are the headers and network info I pulled from chrome:
Remote Address: REMOVED
Request URL: https://REMOVED/default.aspx
Request Method: POST
Status Code: 302 Found
Request Headers:
POST /default.aspx HTTP/1.1
Host: REMOVED
Connection: keep-alive
Content-Length: 928
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://REMOVED
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://REMOVED/default.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: ASP.NET_SessionId=REMOVED; BIGipServerpool_REMOVED_dmz_80=REMOVED.REMOVED.0000; AUTHCDB=**REMOVED**
Form Data:
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE: /wEP**REMAINDER REMOVED**
__EVENTVALIDATION: /wEd**REMAINDER REMOVED**
jsCheck:
ddlEngine:REMOVED:13008
Username:
Password:
btnLogin.x: 42
btnLogin.y: 9
btnLogin: Login
Response Headers:
Cache-Control: private
Content-Length: 132
Content-Type: text/html; charset=utf-8
Date: Fri, 13 Jun 2014 00:59:13 GMT
Location: /Dashboard.aspx
Server: Microsoft-IIS/7.5
Set-Cookie: AUTHCDB=**REMOVED**; path=/; HttpOnly
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Here is the script I wrote:
import requests
FORM_DATA = {
"__EVENTTARGET:":,
"__EVENTARGUMENT:",
"__VIEWSTATE:/wEPDwUKMTA5NTA5ODU1MQ9kFgJmD2QWAgIGDxBkDxYFZgIBAgICAwIEF***REMAINDER REMOVED***",
"__EVENTVALIDATION:/wEdAAp4d3BHvSTs+Kv6cxGP3xEbBr8xrgRYad2tj4YCyRIw5qUAjimf****REMAINDER REMOVED****",
"jsCheck:",
"ddlEngine: REMOVED:13008",
"Username: ****",
"Password: ****",
"btnLogin.x: 42",
"btnLogin.y: 9",
"btnLogin: Login",
}
HEADER = {
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding":"gzip,deflate,sdch",
"Accept-Language":"en-US,en;q=0.8",
"Cache-Control":"max-age=0",
"Connection":"keep-alive",
"Content-Type":"application/x-www-form-urlencoded",
"Host":"REMOVED",
"Origin":"REMOVED",
"Referer":"REMOVED",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36"
}
LOGIN_URL = "REMOVED"
#requests session to handle cookies.
s = requests.Session()
#Send a POST request with the form data/header info
r = s.post(LOGIN_URL, data=FORM_DATA, headers=HEADER)
if r.status_code == 302:
print "Successfully logged in."
else:
print "Error logging in."
Am I able to use Python Requests to log into a webpage that uses ASP.NET? If so, is this the correct way to pass the credentials into the website? For reference, the website I'm trying to log into is a company server monitor.
Looks like a similar issue was posted here and here. I've been using RoboBrowser and it's made messing with ASPX so much simpler.
from robobrowser import RoboBrowser
login_url = 'http://example.com/Login.aspx'
username = 'JohnDoe'
password = 'passwd'
browser = RoboBrowser(history=True)
# This gets all the ASPX stuff, __VIEWSTATE and friends
browser.open(login_url)
signin = brower.get_form(id='aspnetForm')
signin["jsCheck"].value = ''
signin["ddlEngine"].value = "REMOVED:13008"
signin["Username"].value = username
signin["Password"].value = password
signin["btnLogin.x"].value = "42"
signin["btnLogin.y"].value = "9"
signin["btnLogin"].value = "Login"
browser.submit_form(signin)
I'm also working with some ASP.net pages right now, and being somewhat familiar with the requests module, I thought I'd try to help out a bit.
It is my understanding that requests supports basic authentication in this fashion:
from requests.auth import HTTPBasicAuth
requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
It could be the case that you'll need to import a different authentication library that works with ASP.net and plug that directly into the auth function of requests.
Hope this helps!