I tried to run the Curl command in the terminal. I can get the response in JSON. But when I convert the Curl to Python Code and run the python script, the script returned an html page. It looks like it is a captcha page.
Why they can detect the difference? Any way I can bypass that?
curl --location --request GET 'https://www.99.co/api/v2/web/search/listings?query_type=city&property_segments=residential&listing_type=sale&rental_type=unit&page_size=1&page_num=100&zoom=11&show_future_mrts=true&show_cluster_preview=true&show_internal_linking=true' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0' \
--header 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
--header 'accept-language: en-US,en;q=0.5' \
--header 'if-none-match: W/"5cb28158470b6eb63b0db1f5dd2adda7d10c328c"' \
--header 'cookie: _xsrf=2|6be6c7ee|de091e966dba87e0b2c8637a63127715|1616479899' \
--header 'cookie: lotameid=undefined' \
--header 'cookie: started_since=1616480838' \
--header 'cookie: country=SG' \
--header 'cookie: nid=2de067d4-fed8-4c35-b7db-14f479997f8a' \
--header 'cookie: g_state={"i_p":1627551635034,"i_l":1}' \
--header 'upgrade-insecure-requests: 1' \
--header 'te: trailers' \
--header 'sec-gpc: 1'
import requests
url = "https://www.99.co/api/v2/web/search/listings?query_type=city&property_segments=residential&listing_type=sale&rental_type=unit&page_size=1&page_num=100&zoom=11&show_future_mrts=true&show_cluster_preview=true&show_internal_linking=true"
payload={}
headers = {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'accept-language': 'en-US,en;q=0.5',
'if-none-match': 'W/"5cb28158470b6eb63b0db1f5dd2adda7d10c328c"',
'cookie': '_xsrf=2|6be6c7ee|de091e966dba87e0b2c8637a63127715|1616479899',
'cookie': 'lotameid=undefined',
'cookie': 'started_since=1616480838',
'cookie': 'country=SG',
'cookie': 'nid=2de067d4-fed8-4c35-b7db-14f479997f8a',
'cookie': 'g_state={"i_p":1627551635034,"i_l":1}',
'upgrade-insecure-requests': '1',
'te': 'trailers',
'sec-gpc': '1'
}
response = requests.request("GET", url, headers=headers, data=payload)
print(response.text)
Related
The site that I am trying to scrape must follow 302 observed by the browser network tools. I've copied the network request as Curl and its working fine, but when I convert it to python requests its just returning 200.
Curl:
curl -v 'https://my-site.com/CardAuthentication.aspx' \
-H 'Connection: keep-alive' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'Origin: https://my-site.com' \
-H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBN2XwBbcAwvRWZzk' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-GPC: 1' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Referer: https://my-site.com/CardAuthentication.aspx' \
-H 'Accept-Language: en-US,en;q=0.9' \
-H 'Cookie: ASP.NET_SessionId=...; TS0192fa71=...; ServiceProvider=UID=...; .ASPXAUTH=...' \
--data-raw $'------WebKitFormBoundaryBN2XwBbcAwvRWZzk\r\ DATA \r\n' \
--compressed
this is returning:
* Trying IP:443...
........
........
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: my-site.com/MemberDetails.aspx
< Date: Mon, 07 Feb 2022 09:54:49 GMT
< Content-Length: 61211
< Set-Cookie: TS0192fa71=...; Path=/; Domain=.my-site.com; Secure
Python:
import requests
import logging
logging.basicConfig(level=logging.DEBUG)
cookies = {
"ASP.NET_SessionId": ".....",
"TS0192fa71": ".....",
"ServiceProvider": ".....",
".ASPXAUTH": ".....",
}
headers = {
......
"Content-Type": "multipart/form-data; boundary=----WebKitFormBoundaryBN2XwBbcAwvRWZzk",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
"Referer": "https://my-site.com/CardAuthentication.aspx",
"Accept-Language": "en-US,en;q=0.9",
}
data = {
"------WebKitFormBoundaryBN2XwBbcAwvRWZzk\r\nContent-Disposition: form-data; name": '"__EVENTTARGET"\r\n\r\n\r\n-- DATA
}
response = requests.post(
"https://my-site.com/CardAuthentication.aspx",
headers=headers,
cookies=cookies,
data=data,
)
And the return code is 200, with response history empy.
Is there something wrong with requests library or in the way that request library is processing the data? How can I solve this?
Is there something wrong with requests library or in the way that
request library is processing the data? How can I solve this?
This is default requests behavior, you need to set allow_redirects to False if you want to not follow in case of 3xx response code, example
import requests
r = requests.get("http://github.com", allow_redirects=False)
print(r.status_code) # 301
If you want to know more read request.request docs.
I am trying to implement a cURL request into Swift App, however, the server is not accepting my request for some reason.
cURL Request:
curl 'https://test.com' \
-H 'Connection: keep-alive' \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'cache-control: no-cache' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36' \
-H 'Content-Type: application/json' \
-H 'Origin: https://yyy.com' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Referer: https://zzz.com' \
-H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
--data-binary '{"method":"NewUser","params":{"Username":"James","Password":"secret"}}' \
--compressed
I have managed to do this in Python and it worked:
import requests
url = "https://test.com"
headers = CaseInsensitiveDict
headers["Connection"] = "keep-alive"
headers["Accept"] = "application/json, text/javascript, */*; q=0.01"
headers["cache-control"] = "no-cache"
headers["X-Requested-With"] = "XMLHttpRequest"
headers["User-Agent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
headers["Content-Type"] = "application/json"
headers["Origin"] = "https://yyy.com"
headers["Sec-Fetch-Site"] = "same-origin"
headers["Sec-Fetch-Mode"] = "cors"
headers["Sec-Fetch-Dest"] = "empty"
headers["Referer"] = "https://zzz"
headers["Accept-Language"] = "en-GB,en-US;q=0.9,en;q=0.8"
data = '{"method":"NewUser","params":{"Username":"James","Password":"Secret"}}'
raw_response = requests.post(url, headers=headers, data=data)
What I am trying to do in Swift is:
let url = URL(string: "https://test.com")!
var request = URLRequest(url: url)
request.httpMethod = "POST"
request.addValue("application/json", forHTTPHeaderField: "Content-Type")
request.addValue("keep-alive", forHTTPHeaderField: "Connection")
request.addValue("application/json, text/javascript, */*; q=0.01", forHTTPHeaderField: "Accept")
request.addValue("no-cache", forHTTPHeaderField: "cache-control")
request.addValue("XMLHttpRequest", forHTTPHeaderField: "X-Requested-With")
request.addValue("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36", forHTTPHeaderField: "User-Agent")
request.addValue("https://yyy.com", forHTTPHeaderField: "Origin")
request.addValue("same-origin", forHTTPHeaderField: "Sec-Fetch-Site")
request.addValue("cors", forHTTPHeaderField: "Sec-Fetch-Mode")
request.addValue("empty", forHTTPHeaderField: "Sec-Fetch-Dest")
request.addValue("https://zzz.com", forHTTPHeaderField: "Referer")
request.addValue("en-GB,en-US;q=0.9,en;q=0.8", forHTTPHeaderField: "Accept-Language")
let json: [String: Any] = ["method": "NewUser","params": ["Username":"James", "Password":"Secret"]]
let jsonData = try! JSONSerialization.data(withJSONObject: json, options: [])
request.httpBody = jsonData
let task = URLSession.shared.dataTask(with: url) {(data, response, error) in
guard let data = data else { return }
print(String(data: data, encoding: .utf8)!)
}
task.resume()
Any idea what is the difference between Python/Swift code that it's not working?
Thanks.
I'm currently getting forbidden for a site I am trying to visit using requests library.
What's interesting is it seems to be working fine when using curl like this
curl 'https://www.zim.com/tools/track-a-shipment?consnumber=ZIMUHKG83103991' \
-H 'Connection: keep-alive' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'sec-ch-ua: "Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Referer: https://www.zim.com/tools/track-a-shipment?consnumber=ZIMUHKG83103991' \
-H 'Accept-Language: en-US,en;q=0.9' \
-H 'Cookie: bm_sz=A9168FBFA535588B6FA41B389B0E0FC1~YAAQ5zLFF9vG5S52AQAA4DyEaQroeTnfUz1nzswWQ4xFrGAXCczEwMD0MIS+Fr9nA5Zs5CDuB/6MOUP/Q8maKyJJN1yTDoeodUjZt+5T/loHrw8YlbK3a0vO+29JspP04oee8T5lU9qct2gtHC60vbG94f3quPmuSVBERJbEnOG/Ju5QoQx9jsDMPQ==; AKA_A2=A; ak_bmsc=F05A9B88D5CE9314962A4FB53AC066FE17C532E756050000F179D95F94CDFE71~plTDPprwDhfZXi2vSQbthcJGHAoK6c1KvvX8PW2BvX9/io3C7PDJwH1q8MiYF2HN4xoooBKQWo3LvCfAKOWQaTOhy3Q+2auRG/2mn2fLiJxvIuju5c4huzOMGsxmM1BrVtWKf+fR4nnfdtodyimozv+SyFazbEbuHuq18ReiUyoP5KyXXlrsRkp3ReYMkve37Mm6n4x/8s7ZWSa7scVVthWDn02d9Dtp4buykLxcZCz1s=; rxVisitor=16080880493048D6AKB20IDEQSEGR1UGPHIO59J02C95B; uniqueUserId=73aa42f6-ec0a-454b-864c-53c929c69acc; stg_traffic_source_priority=1; country=US; _fbp=fb.1.1608088050866.874878324; _ga=GA1.2.1801289879.1608088051; _gid=GA1.2.1440505168.1608088051; _pk_ses.923228e7-d119-45eb-a025-e4d1b1af6e1b.ad39=*; OptanonAlertBoxClosed=2020-12-16T03:07:34.685Z; dtCookie=v_4_srv_1_sn_5RCLUTEDABIFNF5FLA270N8NCOHUOMRF_perc_100000_ol_0_mul_1_app-3A25bc709b3f2362bb_0; TS0128f558=01bbeceaf7fc47a22dbac4f704c3f650720bf639f1b2b937dd7133075e6b113a96aa9c1b98d65581f9a2ca694795bd9b5c3403231723523f99d38545aaef94cf6cc57266cf; LPVID=MwZTliY2MwYjcwMDFmOTdi; LPSID-53134936=JThIkUaOQaGDwAiVk_mZWg; TS01f7aa8d=01bbeceaf7e716ede662d1632457f3471645d66c1808274488c7c06b899e5092889d3e65ade3ba07fd262b9063dd8f63ed770f2bef; stg_externalReferrer=https://www.zim.com/tools/track-a-shipment?consnumber=ZIMUHKG83103991; _abck=7AE32D3F4822078BCDBCB745422E19AE~0~YAAQ5zLFF4XV5S52AQAAOkGOaQUBzX68zlTJTRs1RP+1RohmAGbj05cNVwmJZIYznbQcOmT6NR0ccyXvbkmcczETVxvbyg4h9qiNQ9TNDh9aZrnqxQxttgA96/eKOY5iysj2qIM5Fg+RCBqlSS96tse+yduqQ+JV2OKFCgQC0CR+5Fvr8nngGjIGVqsV/wC4vNGBQX6oqf1YHMxx8rN36mEzTfOpQvUAVU1iN0fuK98Rpu16LFgJU05GRxd/trtTYKDx8AX6SSdjpGyNIvJWOKg30wO7ALYMjALr3pPiWh23sF6McV3nOJVONoGZpBrK6bhEOfQAeUnqkYT4aySK1JebTA==~-1~-1~-1; dtLatC=37; OptanonConsent=isIABGlobal=false&datestamp=Tue+Dec+15+2020+19%3A18%3A19+GMT-0800+(Pacific+Standard+Time)&version=5.14.0&landingPath=NotLandingPage&groups=1%3A1%2C2%3A1&AwaitingReconsent=false; _pk_id.923228e7-d119-45eb-a025-e4d1b1af6e1b.ad39=e4e8f967bd0568d5.1608088052.1.1608088700.1608088052.; bm_sv=DA4D33FF58BFFFC86A150B9B747988EC~oo1jX8ygs4LrnU4muVv/GaJaPJk2Jiolx5z0YcxGR4jlocdv9gImCEfeD3qrIbuauRTjMQYhGgV9HfY6u+d2QkpVguGGBIc92tToxRY4Ht3kXG+UqE5NpoBKiBhweyNtt/LIFRgTDDA1v4zQc8EkBw==; rxvt=1608090500507|1608088049306; dtPC=1$88698827_281h-vKTCJGFJVHKAFAPRGCSPMAVPCDVLMHPUJ-0e8; dtSa=true%7CC%7C-1%7CTrack%20Shipment%7C-%7C1608089388256%7C88698827_281%7Chttps%3A%2F%2Fwww.zim.com%2Ftools%2Ftrack-a-shipment%3Fconsnumber%3DZIMUHKG83103991%7CTrack%20Shipment%5Ec%20Container%20Tracing%20%5Ep%20ZIM%7C1608088700148%7C%7C; RT="sl=4&ss=1608088049148&tt=11039&obo=0&sh=1608088700486%3D4%3A0%3A11039%2C1608088668289%3D3%3A0%3A9023%2C1608088063737%3D2%3A0%3A5494%2C1608088051968%3D1%3A0%3A2818&dm=zim.com&si=puidioyg8d8&rl=1&ld=1608088700486&nu=&cl=1608089388285&r=https%3A%2F%2Fwww.zim.com%2Ftools%2Ftrack-a-shipment%3F201a15c26295707f7d9507ad6c103a3f&ul=1608089388315"; stg_last_interaction=Wed%2C%2016%20Dec%202020%2003:29:48%20GMT; stg_returning_visitor=Wed%2C%2016%20Dec%202020%2003:29:48%20GMT' \
--compressed
Doing the same exact thing in request and making sure I am passing every header correctly, the site still seem to give me forbidden error. I understand this is done for security on their side, but it seems to be working fine using curl, what could be different?
My python requests look like this:
headers = {
'Connection': 'keep-alive',
'Pragma': 'no-cache',
'Cache-Control': 'no-cache',
'sec-ch-ua': '"Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87"',
'sec-ch-ua-mobile': '?0',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Sec-Fetch-Site': 'same-origin',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-User': '?1',
'Sec-Fetch-Dest': 'document',
'Referer': 'https://www.zim.com/tools/track-a-shipment',
'Accept-Language': 'en-US,en;q=0.9',
'Cookie':'bm_sz=A9168FBFA535588B6FA41B389B0E0FC1~YAAQ5zLFF9vG5S52AQAA4DyEaQroeTnfUz1nzswWQ4xFrGAXCczEwMD0MIS+Fr9nA5Zs5CDuB/6MOUP/Q8maKyJJN1yTDoeodUjZt+5T/loHrw8YlbK3a0vO+29JspP04oee8T5lU9qct2gtHC60vbG94f3quPmuSVBERJbEnOG/Ju5QoQx9jsDMPQ==; AKA_A2=A; ak_bmsc=F05A9B88D5CE9314962A4FB53AC066FE17C532E756050000F179D95F94CDFE71~plTDPprwDhfZXi2vSQbthcJGHAoK6c1KvvX8PW2BvX9/io3C7PDJwH1q8MiYF2HN4xoooBKQWo3LvCfAKOWQaTOhy3Q+2auRG/2mn2fLiJxvIuju5c4huzOMGsxmM1BrVtWKf+fR4nnfdtodyimozv+SyFazbEbuHuq18ReiUyoP5KyXXlrsRkp3ReYMkve37Mm6n4x/8s7ZWSa7scVVthWDn02d9Dtp4buykLxcZCz1s=; rxVisitor=16080880493048D6AKB20IDEQSEGR1UGPHIO59J02C95B; uniqueUserId=73aa42f6-ec0a-454b-864c-53c929c69acc; stg_traffic_source_priority=1; country=US; _fbp=fb.1.1608088050866.874878324; _ga=GA1.2.1801289879.1608088051; _gid=GA1.2.1440505168.1608088051; _pk_ses.923228e7-d119-45eb-a025-e4d1b1af6e1b.ad39=*; OptanonAlertBoxClosed=2020-12-16T03:07:34.685Z; dtCookie=v_4_srv_1_sn_5RCLUTEDABIFNF5FLA270N8NCOHUOMRF_perc_100000_ol_0_mul_1_app-3A25bc709b3f2362bb_0; TS0128f558=01bbeceaf7fc47a22dbac4f704c3f650720bf639f1b2b937dd7133075e6b113a96aa9c1b98d65581f9a2ca694795bd9b5c3403231723523f99d38545aaef94cf6cc57266cf; LPVID=MwZTliY2MwYjcwMDFmOTdi; LPSID-53134936=JThIkUaOQaGDwAiVk_mZWg; TS01f7aa8d=01bbeceaf7e716ede662d1632457f3471645d66c1808274488c7c06b899e5092889d3e65ade3ba07fd262b9063dd8f63ed770f2bef; stg_externalReferrer=https://www.zim.com/tools/track-a-shipment?consnumber=ZIMUHKG83103991; _abck=7AE32D3F4822078BCDBCB745422E19AE~0~YAAQ5zLFF4XV5S52AQAAOkGOaQUBzX68zlTJTRs1RP+1RohmAGbj05cNVwmJZIYznbQcOmT6NR0ccyXvbkmcczETVxvbyg4h9qiNQ9TNDh9aZrnqxQxttgA96/eKOY5iysj2qIM5Fg+RCBqlSS96tse+yduqQ+JV2OKFCgQC0CR+5Fvr8nngGjIGVqsV/wC4vNGBQX6oqf1YHMxx8rN36mEzTfOpQvUAVU1iN0fuK98Rpu16LFgJU05GRxd/trtTYKDx8AX6SSdjpGyNIvJWOKg30wO7ALYMjALr3pPiWh23sF6McV3nOJVONoGZpBrK6bhEOfQAeUnqkYT4aySK1JebTA==~-1~-1~-1; dtLatC=37; OptanonConsent=isIABGlobal=false&datestamp=Tue+Dec+15+2020+19%3A18%3A19+GMT-0800+(Pacific+Standard+Time)&version=5.14.0&landingPath=NotLandingPage&groups=1%3A1%2C2%3A1&AwaitingReconsent=false; _pk_id.923228e7-d119-45eb-a025-e4d1b1af6e1b.ad39=e4e8f967bd0568d5.1608088052.1.1608088700.1608088052.; bm_sv=DA4D33FF58BFFFC86A150B9B747988EC~oo1jX8ygs4LrnU4muVv/GaJaPJk2Jiolx5z0YcxGR4jlocdv9gImCEfeD3qrIbuauRTjMQYhGgV9HfY6u+d2QkpVguGGBIc92tToxRY4Ht3kXG+UqE5NpoBKiBhweyNtt/LIFRgTDDA1v4zQc8EkBw==; rxvt=1608090500507|1608088049306; dtPC=1$88698827_281h-vKTCJGFJVHKAFAPRGCSPMAVPCDVLMHPUJ-0e8; dtSa=true%7CC%7C-1%7CTrack%20Shipment%7C-%7C1608089388256%7C88698827_281%7Chttps%3A%2F%2Fwww.zim.com%2Ftools%2Ftrack-a-shipment%3Fconsnumber%3DZIMUHKG83103991%7CTrack%20Shipment%5Ec%20Container%20Tracing%20%5Ep%20ZIM%7C1608088700148%7C%7C; RT="sl=4&ss=1608088049148&tt=11039&obo=0&sh=1608088700486%3D4%3A0%3A11039%2C1608088668289%3D3%3A0%3A9023%2C1608088063737%3D2%3A0%3A5494%2C1608088051968%3D1%3A0%3A2818&dm=zim.com&si=puidioyg8d8&rl=1&ld=1608088700486&nu=&cl=1608089388285&r=https%3A%2F%2Fwww.zim.com%2Ftools%2Ftrack-a-shipment%3F201a15c26295707f7d9507ad6c103a3f&ul=1608089388315"; stg_last_interaction=Wed%2C%2016%20Dec%202020%2003:29:48%20GMT; stg_returning_visitor=Wed%2C%2016%20Dec%202020%2003:29:48%20GMT'
}
params = (
('consnumber', 'ZIMUHKG83103991'),
)
response = requests.get('https://www.zim.com/tools/track-a-shipment', headers=headers, params=params)
I even make sure to pass the same exact cookie, it still fails.
you can change
params = (
('consnumber', 'ZIMUHKG83103991'),
)
to dict type params = {
'consnumber':'ZIMUHKG83103991'
} try again
It is bizarre activity, I get different response from cURL and Python with the requests module. Also, Python is very slow: cURL takes about 6 seconds, but Python is 2 minutes.
My cURL command is:
curl -L -s -k --request GET \
--url "https://$1/api/maintenance/maintenance/dual_image_config" \
--header 'Cache-Control: no-cache' \
--header 'Accept-Encoding: gzip, deflate, sdch' \
--header 'Content-Type: application/json' \
--header 'Connection: keep-alive'
The cURL response is:
{ "state_1": "Active", "state_2": "stand-by", "current_active_image": "Image-1", "id": 1}
My Python code is:
import requests, sys
DEFAULT_HEADER = {"Accept-Encoding": "gzip, deflate, br",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"Content-Type": "application/json",
"X-Requested-With": "XMLHttpRequest"}
url = "https://{0}/api/maintenance/maintenance/dual_image_config".format(sys.argv[1])
response = requests.get(url="", data=None, headers=DEFAULT_HEADER , verify=False)
Python response response.text is:
'0121\r\n{ "state_1": "Active", "state_2": "stand-by", "current_active_image": "Image-1", "id": 1}\r\n0\r\n\r\n'
It will have 0121\r\n & \r\n0\r\n\r\n in start and begin, I have try to change request header & encoding, but it not work, why it is got different response, and so slow ?
Here is the curl command:
curl 'http://ecard.bupt.edu.cn/User/ConsumeInfo.aspx'
-H 'Cookie: ASP.NET_SessionId=4mzmi0whscqg4humcs5fx0cf; .NECEID=1; .NEWCAPEC1=$newcapec$:zh-CN_CAMPUS; .ASPXAUTSSM=A91571BB81F620690376AF422A09EEF8A7C4F6C09978F36851B8DEDFA56C19C96A67EBD8BBD29C385C410CBC63D38135EFAE61BF49916E0A6B9AB582C9CB2EEB72BD9DE20D64A51C09474E676B188D937B84E601A981C02F51AA9C85A08EABCC1D30D7B9299E02A45D427B08A44AEBB0'
-H 'Origin: http://ecard.bupt.edu.cn'
-H 'Accept-Encoding: gzip, deflate'
-H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4'
-H 'Upgrade-Insecure-Requests: 1'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Cache-Control: max-age=0' -H 'Referer: http://ecard.bupt.edu.cn/User/ConsumeInfo.aspx'
-H 'Connection: keep-alive'
--data '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUINDEyMzA5NDkPFgIeCFNvcnRUeXBlBQNBU0MWAmYPZBYCAgMPZBYCAgMPZBYCAgQPPCsAEQMADxYEHgtfIURhdGFCb3VuZGceC18hSXRlbUNvdW50ZmQBEBYAFgAWAAwUKwAAFgJmD2QWAmYPZBYCAgEPZBYCAgEPDxYEHghDc3NDbGFzcwUKU29ydEJ0X0FzYx4EXyFTQgICZGQYAQUiY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRncmlkVmlldw88KwAMAQhmZBb11RlvPBlQn5fbuqe6uh8BRZJ2jUZ5U17xEnRM%2BnbF&__VIEWSTATEGENERATOR=D99777FC&__EVENTVALIDATION=%2FwEdAAgtbl4l0SdCQqYpyy3Ex39BEoHXgPeD%2Fa3eEKPr2bG0rY8WyuyQajjUbeopYM%2Fne68pyn2l0BPEWPPnyd6MfoDZVQGbFRIKjb%2FOTPkkCDIIaJ3X6W3VKO%2FV036Pdf6jf06OTFulzGhY80%2FZ1HbCJJ6LR5Lg6mLp72ibUCB3VipRJuK11qmW%2BSDe3IOvbK4oNUdV5%2FxmkXw25tTwfJ8P8OS2&ctl00%24ContentPlaceHolder1%24rbtnType=0&ctl00%24ContentPlaceHolder1%24txtStartDate=2016-09-17&ctl00%24ContentPlaceHolder1%24txtEndDate=2016-09-24&ctl00%24ContentPlaceHolder1%24btnSearch=%E6%9F%A5++%E8%AF%A2' --compressed
I want to use request to get the return message to analyse, but i'm confused about this.
Thanks!
You can use python requests module. The answer would be something like:
response = requests.get('http://ecard.bupt.edu.cn/User/ConsumeInfo.aspx',
headers={'Origin': 'http://ecard.bupt.edu.cn'},
data={'__EVENTTARGET': '', 'rbtnType':0},
cookies={'.NECEID': 1}
)
I didn't put all parameters there but I think this example is enough to figure out how to send exact the same request you want. The only thing I am not sure is --compressed flag - as I understand from docs it is something more specific to curl(correct me if I'm wrong)
Here is a useful website to convert the curl to requests:
curl to request
Curl Command:
curl 'http://js.t.sinajs.cn/open/api/js/api/bundle.js?version=20150130.02' -H 'Pragma: no-cache' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.87 Safari/537.36' -H 'Accept: */*' -H 'Referer: http://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/001375502200090e998439175cc4268b0ea703b3b4ed55e000' -H 'Connection: keep-alive' -H 'Cache-Control: no-cache' --compressed
Python requests:
import requests
headers = {
'Pragma': 'no-cache',
'Accept-Encoding': 'gzip, deflate, sdch',
'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.87 Safari/537.36',
'Accept': '*/*',
'Referer': 'http://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/001375502200090e998439175cc4268b0ea703b3b4ed55e000',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache',
}
requests.get('http://js.t.sinajs.cn/open/api/js/api/bundle.js?version=20150130.02', headers=headers)