Develop Reference

python post request with cookies asp.net

Please help. I'm trying to create a POST request on an .asp site that requires cookies, but the way I handle them seems not to return anything. Read through some questions of similar topic but can't find the _SessionID cookie some are referring to. Please help me formulate this POST request so it works.
Headers
:authority: safer.fmcsa.dot.gov
:method: POST
:path: /query.asp
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
content-length: 85
content-type: application/x-www-form-urlencoded
cookie: ASP.NET_SessionId=ywxszihqlu1yciwe5z5gm4qt; etype=au; ASPSESSIONIDQECTCDRB=KGFOBHBBLKCBKBFIAPEBMIHJ; ASPSESSIONIDQGARCDRA=LKEDBAOBMOMDNGBNBFEMMIPB; ASPSESSIONIDSEBQCBSB=DAMJMNKCNJKHCMDCIJBPKEHD; ASPSESSIONIDCEQRADQC=EIEJCDLBHHCCKHCNJNIMHDKA; ASPSESSIONIDAESTCBQC=KPDPJNHCLOBJENEHPNIFKJLH; LI_carrier=67449; ASPSESSIONIDAGSQADRC=CPIBAKEDDPNFCIPLIGLOKKLA; ASPSESSIONIDAERTDBQD=FMKFHJJAJKNIGCBCCFJFCMNF; AWSALB=Xc7OAuZUmx6vgE5l9NaawsH8oBWjy6eZ3B62kw2rZ5HieoRlMu4SSmVVcaPJPcPjp1fVt9U/T9FaRflgNHwtzmsK4X4e+y+yoGArTfgpb75NWo/ilAek0Qk/sFYI; AWSALBCORS=Xc7OAuZUmx6vgE5l9NaawsH8oBWjy6eZ3B62kw2rZ5HieoRlMu4SSmVVcaPJPcPjp1fVt9U/T9FaRflgNHwtzmsK4X4e+y+yoGArTfgpb75NWo/ilAek0Qk/sFYI
origin: https://safer.fmcsa.dot.gov
referer: https://safer.fmcsa.dot.gov/CompanySnapshot.aspx
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36
Form Data
searchtype: ANY
query_type: queryCarrierSnapshot
query_param: USDOT
query_string: 2300842
My Code So Far
def checkDOT():
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Language': 'en-US,en;q=0.9',
'Cache-Control': 'no-cache,no-store,must-revalidate,max-age=0,private',
'upgrade-insecure-requests': '1',
'Connection': 'keep-alive',
'origin': 'https://safer.fmcsa.dot.gov',
'referer': 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx'
}
s = requests.Session()
data = {
'searchtype': 'ANY',
'query_type': 'queryCarrierSnapshot',
'query_param': 'USDOT',
'query_string': '2300842'
}
params = (
('pageNumber', '0'),
('itemsPerPage', '15'),
)
url = 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx'
response = s.get(url, headers=headers, data=data, params=params)
if response:
print(response.content)
else:
print("This did not work")

get requests dont use data parameter, and your code is requests.get,is that right?
I can get a html page with:
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0',
}
url = 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx'
response = requests.get(url, headers=headers, verify=False)
print(response.text)

How to send a GET request with headers via python

I got fiddler to capture a GET request, I want to re send the exact request with python.
This is the request I captured:
GET https://example.com/api/content/v1/products/search?page=20&page_size=25&q=&type=image HTTP/1.1
Host: example.com
Connection: keep-alive
Search-Version: v3
Accept: application/json
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
Referer: https://example.com/search/?q=&type=image&page=20
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

You can use the requests module.
The requests module automatically supplies most of the headers for you so you most likely do not need to manually include all of them.
Since you are sending a GET request, you can use the params parameter to neatly form the query string.
Example:
import requests
BASE_URL = "https://example.com/api/content/v1/products/search"
headers = {
"Connection": "keep-alive",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
}
params = {
"page": 20,
"page_size": 25,
"type": "image"
}
response = requests.get(BASE_URL, headers=headers, params=params)

import requests
headers = {
'authority': 'stackoverflow.com',
'cache-control': 'max-age=0',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'referer': 'https://stackoverflow.com/questions/tagged/python?sort=newest&page=2&pagesize=15',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9,tr-TR;q=0.8,tr;q=0.7',
'cookie': 'prov=6bb44cc9-dfe4-1b95-a65d-5250b3b4c9fb; _ga=GA1.2.1363624981.1550767314; __qca=P0-1074700243-1550767314392; notice-ctt=4%3B1550784035760; _gid=GA1.2.1415061800.1552935051; acct=t=4CnQ70qSwPMzOe6jigQlAR28TSW%2fMxzx&s=32zlYt1%2b3TBwWVaCHxH%2bl5aDhLjmq4Xr',
}
response = requests.get('https://stackoverflow.com/questions/55239787/how-to-send-a-get-request-with-headers-via-python', headers=headers)
This is an example of how to send a get request to this page with headers.

You may open SSL socket (https://docs.python.org/3/library/ssl.html) to example.com:443, write your captured request into this socket as raw bytes, and then read HTTP response from the socket.
You may also try to use http.client.HTTPResponse class to read and parse HTTP response from your socket, but this class is not supposed to be instantiated directly, so some unexpected obstacles could emerge.

Permission denied when using Python request session to make multiple requests

I am trying to automate 2 requests using python, first request is GET and the 2nd is POST
Here is how I manually do it using Chrome,
I visit http://testserver/index on chrome browser.
It prompt me for NTLM login. I provided username/password which was successful. Then I went on another page,
http://testserver/find_user and enter a username to search for. I press entered which displayed the results.
Then I copy the curl request from Chrome, convert it into python code and got this,
import requests
with requests.Session() as session:
session.auth = HttpNtlmAuth("DOSTR\\TESTUSER", getpass.getpass('Password:'))
url = "http://testserver/find_user"
payload = "username=test"
headers = {
'Connection': "keep-alive",
'Cache-Control': "max-age=0",
'Origin': "http://testserver",
'Upgrade-Insecure-Requests': "1",
'Content-Type': "application/x-www-form-urlencoded",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
'Referer': "http://testserver/find_user?thread=2&aftk-687=-719740030",
'Accept-Encoding': "gzip, deflate",
'Accept-Language': "en-GB,en-US;q=0.9,en;q=0.8",
'Cookie': "JSESSIONID=4D8270489027BCD04777AAB32769B3A9; lang=en; mode=index"
}
response = session.request("POST", url, data=payload, headers=headers)
print(response.text)
The above request works.
But the problem is I have to make a first request using Chrome to generate cookies.
So I tried making the first request using Python as well like this, and use its cookies in the 2nd request
with requests.Session() as session:
session.auth = HttpNtlmAuth("DOSTR\\TESTUSER", getpass.getpass('Password:'))
url = "http://testserver/index"
headers = {
'Connection': "keep-alive",
'Upgrade-Insecure-Requests': "1",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
'Accept-Encoding': "gzip, deflate",
'Accept-Language': "en-GB,en-US;q=0.9,en;q=0.8"
}
response = session.request("GET", url, headers=headers, allow_redirects=True)
# now 2nd request in same session with cookies of above response.
url = "http://testserver/find_user"
payload = "username=test"
headers = {
'Connection': "keep-alive",
'Cache-Control': "max-age=0",
'Origin': "http://testserver",
'Upgrade-Insecure-Requests': "1",
'Content-Type': "application/x-www-form-urlencoded",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
'Referer': "http://testserver/find_user?thread=2&aftk-687=-719740030",
'Accept-Encoding': "gzip, deflate",
'Accept-Language': "en-GB,en-US;q=0.9,en;q=0.8"
}
response = session.request("POST", url, data=payload, headers=headers, cookies=response.cookies)
print(response.text)
But I keep getting permission denied error on the 2nd request (The GET request is successful, and I can see the output if I print it)
it only works when I use the cookies in 2nd request generated by Chrome, but not when I generate those cookies using python
I am not sure why the cookies from first request is not working in the 2nd request.
Can someone please tell me what am I doing wrong?
Edit:
Response header from GET request in chrome,
HTTP/1.1 200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Type: text/html;charset=UTF-8
Content-Encoding: gzip
Expires: Sat, 6 May 1995 12:00:00 GMT
Server: Microsoft-IIS/7.5
X-Frame-Options: DENY
Set-Cookie: JSESSIONID=F8DC91356195C0D1730638B81A60F6EB; Path=/index/; HttpOnly
Set-Cookie: lang=en; Expires=Mon, 09-Apr-2068 18:49:54 GMT
Persistent-Auth: true
X-Powered-By: ASP.NET
Date: Tue, 05 Feb 2019 21:24:57 GMT
Content-Length: 13267
Response header from GET request in Python,
{'Cache-Control': 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Pragma': 'no-cache', 'Content-Length': '984', 'Content-Type': 'text/html;charset=UTF-8',
'Content-Encoding': 'gzip', 'Expires': 'Sat, 6 May 1995 12:00:00 GMT', 'Server': 'Microsoft-IIS/7.5', 'X-Frame-Options': 'DENY', 'Set-Cookie': 'JSESSIONID=EF3589A5EC319542C6254C16418F6265; Path=/index/; HttpOnly', 'Persistent-Auth': 'true', 'X-Powered-By': 'ASP.NET', 'Date': 'Tue, 05 Feb 2019 21:27:33 GMT'}

Login with Requests - Not capturing redirects

I am trying to learn requests module and was practice logging to a website but for some reason the it is not working and i am unable to login.
import requests
import sys
param1 = sys.argv[1]
param2 = sys.argv[2]
url2 = 'https://myhpgas.in/myHPGas/Login.aspx'
with requests.Session() as s:
s.get(url2)
print(r.headers)
payload = {'ctl00$ContentPlaceHolder1$txtUserNameEmail': param1,
'ctl00$ContentPlaceHolder1$txtPassword': param2}
p = s.post(url2, data=payload, headers=headers)
p.raise_for_status()
r = s.get('https://myhpgas.in/myHPGas/HPGas/User/ConsumerConsole.aspx')
print(r.text)
Session cookie.
{'ARRAffinity': 'ab2cda67a33c1a756e728834a3f88bc425b66b583804aee440e53c204539d683'}
Request headers for POST request
{'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'ARRAffinity=ab2cda67a33c1a756e728834a3f88bc425b66b583804aee440e53c204539d683', 'Content-Length': '125', 'Content-Type': 'application/x-www-form-urlencoded'}
Response headers for POST request
{'Cache-Control': 'private', 'Content-Length': '7404', 'Content-Type': 'text/html; charset=utf-8', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'Set-Cookie': '.CZONEAUTH=; expires=Mon, 11-Oct-1999 18:30:00 GMT; path=/; HttpOnly', 'X-AspNet-Version': '4.0.30319', 'X-Powered-By': 'ASP.NET, ARR/2.5, ASP.NET', 'Date': 'Fri, 05 May 2017 13:49:59 GMT
This is how form data from browser looks like
tsmManager_HiddenField:
__EVENTTARGET:
__EVENTARGUMENT:
__LASTFOCUS:
__VIEWSTATE:/wEPDwULLTEwODE4NjM1MzcPZBYCZg9kFgICBA9kFgwCAQ8QZGQWAWZkAgUPFgIeB1Zpc2libGVoZAIGDxYCHwBoZAIJDw8WAh4EVGV4dGVkZAILDxYCHwBnZAIMD2QWAmYPZBYCAgUPZBYCAgMPZBYCAgEPZBYCAgEPZBYCAgEPZBYEAgMPFgIeBXN0eWxlBbEBYmFja2dyb3VuZDp1cmwoQ29udHJvbHMvR2VuZXJhdGVDYXB0Y2hhLmFzcHg/NjM2Mjk2MDE0MjUxMTMzNTI2KSBuby1yZXBlYXQ7YmFja2dyb3VuZC1zaXplOjEwMHB4IDMwcHg7aGVpZ2h0OjMwcHg7d2lkdGg6MTAwcHg7YmFja2dyb3VuZC1zaXplOjEwMHB4IDMwcHg7aGVpZ2h0OjMwcHg7d2lkdGg6MTAwcHg7ZAIFDw8WAh8BBVM8aW1nIHNyYz0nQXBwX1RoZW1lcy9CbHVlL2ltYWdlcy9yZWZyZXNoLnBuZycgYWx0PScnIHN0eWxlPSd2ZXJ0aWNhbC1hbGlnbjogdG9wOycvPmRkZDXNsct5fehr8ckYOVLlGTlg9kZ8dqdv11qCFhUCflOc
__EVENTVALIDATION:/wEdACBL8cV3HeSpiBr5eBlx9SY20Xzuqe7DgFLen1mQ9qwwkgTERbnLJaH6mqTlVXquFl7SgzmX0pXnQBVBYHDsagiMkyN1xWptALXH/a/EZC1F4KWAtKwjmte8a5j5Fhl1Xf6IzLKjB01uqKx6ADvgPGnp98RzIzeJI0XjAUrjE3GioE7EqnQGYNYbWcIn1pAHroN3VSUMn8TK6zgSES6I5dLiG9mftCmBcU0BKKuHa3mUvYxWp7bWYNbw7KFKbXe1cNG4vRds483c4wog6wZz3kG439kSOfHwry7ydkMen+8Oykq3dUvfODXzmhEEomZU6Gpc6cx/Q2ozQ90Cwd74q2EZFSptkDm3zh7+G5RAX6wIBHAtqlGmDyDCRAIiHffjIOz35c1Cw4BBvDKiEHQ47ZVnq++zVWerHmq9rIPE2hYysPySrcngv4yAvO4y9I26+ioOlSP/t2GtC9XJw3/Csh53EA1Y7qClBJnaD+WR8QqXtMM2zR3WYG3GJBTLRhXO2+5Z/ephgfX47bkgLErB0CQD3W66Z0s6zx7FcGFEl6h5gfEAYrHaepPURBQevSsriBousIddxfsdtnH74OnL9jhz83qJSb3w8Gz5FiWFgonTQHjjswiyhm+g6KodwobC/fOwS4xnW8ZIVB06X7opsHDkeSC1J6i9rb2nVg34wXRSxTpegcArnmtEi7N5qZc9BFbIJWQgX02aZk1hcI5GYiz9
ctl00$ddlSelectLanguage:-1
ctl00$ContentPlaceHolder1$txtUserNameEmail:emailid#domain.com
ctl00$ContentPlaceHolder1$txtPassword:password
ctl00$ContentPlaceHolder1$btnLogin:Login
And below are the request headers that the browser is sending
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:1678
Content-Type:application/x-www-form-urlencoded
Cookie:ARRAffinity=ab2cda67a33c1a756e728834a3f88bc425b66b583804aee440e53c204539d683; _csm_ux_data=; ASP.NET_SessionId=dxonrup25fyldmnwvhuakv4y
Host:myhpgas.in
Origin:https://myhpgas.in
Referer:https://myhpgas.in/myHPGas/Login.aspx
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
So can someone let me know what i am missing here ?

I was able to parse the GET response and then extract the dynamic values from html source and then posted them in POST request, that resolved my issue.

Get HTTP 400 Bad Request when login using Python requests

I'm trying to use requests to log into https://appleid.apple.com/cn (/us should be the same, but get 400 Bad request returned.
session = requests.Session()
productURL = <the URL above>
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, sdch, br",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.6,en;q=0.4",
"Upgrade-Insecure-Requests":"1",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/52.0.2743.116 Chrome/52.0.2743.116 Safari/537.36"
}
session.headers = headers
r = session.get(productURL)
url = "//idmsa.apple.com/appleauth/auth/signin?widgetKey=af1139274f266b22b68c2a3e7ad932cb3c0bbe854e13a79af78dcc73136882c3&language=zh_CN&rv=1"
r = session.get(url)
url = "//idmsa.apple.com/appleauth/auth/signin"
headers = {
"Accept":"application/json, text/javascript, */*; q=0.01",
"Accept-Encoding":"gzip, deflate, br",
"Accept-Language":"zh-CN,zh;q=0.8,zh-TW;q=0.6,en;q=0.4",
"Connection":"keep-alive",
"Content-Length":"77",
"Content-Type":"application/json",
"Host":"idmsa.apple.com",
"Origin":"https://idmsa.apple.com",
"Referer":"//idmsa.apple.com/appleauth/auth/signin?widgetKey=af1139274f266b22b68c2a3e7ad932cb3c0bbe854e13a79af78dcc73136882c3&language=zh_CN&rv=1",
"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/52.0.2743.116 Chrome/52.0.2743.116 Safari/537.36",
"X-Apple-Domain-Id":1,
"X-Apple-I-FD-Client-Info":{"U":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/52.0.2743.116 Chrome/52.0.2743.116 Safari/537.36","L":"en-US","Z":"GMT+08:00","V":"1.1","F":"7da44j1e3NlY5BSo9z4ofjb75PaK4Vpjt4U_98uszHVyVxFAk.lzXJJIneGffLMC7EZ3QHPBirTYKUowRslz8eibjVdxljQlpQJuYY9hte_1an92r5xj6KksmfTPdFdgmVxf7_OLgiPFMJhHFW_jftckkCoqAkCoq4ly_0x0uVMV0jftckcKyAd65hz7fwdGEM6uJ6o6e0T.5EwHXXTSHCSPmtd0wVYPIG_qvoPfybYb5EtCKoxw4EiCvTDfPbJROKjCJcJqOFTsrhsui65KQnK94CaJ6hO3f9p_nH1zDz.ICMpwoNSdqdbAE9XXTneNufuyPBDjaY2ftckuyPB884akHGOg429OMNo71xFmrur.S9RdPQSzOy_Aw7UTlf_0pNA1OXu_Llri5Ly.EKY.6ekL3sdmX.Cr_Jz9KyFxv5icCmVug4WBkl1BQLz4mvmfTT9oaSumKkpjlRiwerbXh8bUu_LzQW5BNv_.BNlYCa1nkBMfs.Byn"},
"X-Apple-Locale":"zh_CN",
"X-Apple-Widget-Key":"af1139274f266b22b68c2a3e7ad932cb3c0bbe854e13a79af78dcc73136882c3",
"X-Requested-With":"XMLHttpRequest"
}
session.headers = headers
payload = {
"accountName" : "accountName",
"password" : "password",
"rememberMe" : False
}
r = session.post(url, params=payload)
Headers info
request headers
{
'Content-Length': '77',
'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.6,en;q=0.4',
'Accept-Encoding': 'gzip, deflate, br',
'X-Apple-I-FD-Client-Info': {
'F': '7da44j1e3NlY5BSo9z4ofjb75PaK4Vpjt4U_98uszHVyVxFAk.lzXJJIneGffLMC7EZ3QHPBirTYKUowRslz8eibjVdxljQlpQJuYY9hte_1an92r5xj6KksmfTPdFdgmVxf7_OLgiPFMJhHFW_jftckkCoqAkCoq4ly_0x0uVMV0jftckcKyAd65hz7fwdGEM6uJ6o6e0T.5EwHXXTSHCSPmtd0wVYPIG_qvoPfybYb5EtCKoxw4EiCvTDfPbJROKjCJcJqOFTsrhsui65KQnK94CaJ6hO3f9p_nH1zDz.ICMpwoNSdqdbAE9XXTneNufuyPBDjaY2ftckuyPB884akHGOg429OMNo71xFmrur.S9RdPQSzOy_Aw7UTlf_0pNA1OXu_Llri5Ly.EKY.6ekL3sdmX.Cr_Jz9KyFxv5icCmVug4WBkl1BQLz4mvmfTT9oaSumKkpjlRiwerbXh8bUu_LzQW5BNv_.BNlYCa1nkBMfs.Byn',
'Z': 'GMT+08:00',
'U': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/52.0.2743.116 Chrome/52.0.2743.116 Safari/537.36',
'L': 'en-US',
'V': '1.1',
},
'Connection': 'keep-alive',
'X-Apple-Widget-Key': 'af1139274f266b22b68c2a3e7ad932cb3c0bbe854e13a79af78dcc73136882c3',
'Origin': '//idmsa.apple.com',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/52.0.2743.116 Chrome/52.0.2743.116 Safari/537.36',
'Host': 'idmsa.apple.com',
'X-Apple-Domain-Id': 1,
'Referer': '//idmsa.apple.com/appleauth/auth/signin?widgetKey=af1139274f266b22b68c2a3e7ad932cb3c0bbe854e13a79af78dcc73136882c3&language=zh_CN&rv=1',
'X-Apple-Locale': 'zh_CN',
'X-Requested-With': 'XMLHttpRequest',
'Content-Type': 'application/json',
}
response headers
{
'X-XSS-Protection': '1; mode=block',
'X-Content-Type-Options': 'nosniff',
'Content-Security-Policy': "default-src *; object-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://*.apple.com https://*.cdn-apple.com; style-src 'unsafe-inline' https://*.apple.com https://*.cdn-apple.com; connect-src 'self'; img-src 'self' data: https://*.apple.com https://*.cdn-apple.com https://*.icloud.com https://*.mzstatic.com; media-src * data:;",
'Content-Encoding': 'gzip',
'Transfer-Encoding': 'chunked',
'Set-Cookie': 'dslang=CN-ZH; Domain=.apple.com; Path=/; Secure; HttpOnly, site=CHN; Domain=.apple.com; Path=/; Secure; HttpOnly',
'Strict-Transport-Security': 'max-age=31536000; includeSubDomains',
'Vary': 'Accept-Encoding',
'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT',
'Server': 'Apple',
'Connection': 'close',
'X-BuildVersion': 'R15',
'Pragma': 'no-cache',
'Cache-Control': 'no-cache, no-store',
'Date': 'Sat, 01 Oct 2016 04:23:19 GMT',
'X-FRAME-OPTIONS': 'DENY',
}
I checked all the headers field with the real request headers, "X-Apple-I-FD-Client-Info" is the only one not correct. Dig a little bit, it was calculated by javascript. 'Z','U','L','V' are constant, depends on you browser info and timezone etc. But the 'F' is a very long random string
Is "X-Apple-I-FD-Client-Info" the problem result in 400 Bad request?
Is this the right way to write something like auto login? By compareing request headers and cookies one by one?
Is it possible to generate or skip header "X-Apple-I-FD-Client-Info"?
How can I get this auto login work?

When you are posting JSON you should use requests like:
r = requests.post(url, json=payload)
also, don't need to hardcode the Content-Length and Content-Type requests package takes care of that.

Since I'm new and can't comment (I don't quite understand the reputation system yet), I'll have to write an answer.
I know that Google recently blocked the login via scripts (well, via most scripts) because it was rather easy to conduct brute force attacks against accounts.
I am presuming that Apple did something very similar and thus making it hard to log onto the AppleId. Do you know for sure that it is possible to login that way?
Greetings,
Narusan