I am trying to scrape the following url as it appears to me in chrome:
http://finance.yahoo.com/q/op?s=MSFT&date=1414713600
For chrome 37 and linux, it appears with the table headers:
Strike, Contract Name, Last, Bid, Ask, Change, %Change, Volume, Open Interest, Implied Volatility
When I grab it with python requests library, it returns a table with the headers.
"Strike Symbol Last Chg Bid Ask Vol Open Interest"
url = "http://finance.yahoo.com/q/op?s=MSFT&date=1414713600"
text = requests.get(url)
Is there any way to use requests to get the data in the same manner as it appears in chrome? I assume I have to send a change in headers, but I have tried a number of different things that have not worked.
my last try was the following:
headers = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding': 'gzip,deflate,sdch',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36',
'content-type': 'application/json'
}
here are the headers chrome sends for me to get the website I want:
GET /q/op?s=MSFT&date=1413590400 HTTP/1.1
Host: finance.yahoo.com
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,es;q=0.6
Cookie: PREF=ID=a6c1e76064f94d39:U=827f4a5f7ace80ef:FF=0:LD=en:TM=1405040931:LM=1410201670:GM=1:S=btbAGy8zLyRsY3Qr; SID=DQAAACwCAADXppdnRkNOk3FSR5EzCt5fvxdXMAH87uz4j_4Xm2aM-zbhU5JEXVw93LhmIjRCf2B-nonvZv4hXuO_V2FYXsdOYCxigM7jpBxhd1RcK-zfzM1w5SrEa6Q5aFfBHvU6p5KkPQ8v4m8srb_k0cxAzf7x5KRlkZ5llB5oDLpiPPUhFORd7y2xobzHvIo9YJ92pv8kmCnPWnNfMUUMK6XEoov56B1SmfJ8XUTyRN3oBIe0MKAPodTznhLlB0_qzqG1ohjEIv_WoiauvWNzmLGuzmQqAuHlMswIBnF5S0ECJ7sb-6Nap9FITjCnJfrelTqTH782wJL9pC7SGlU0L1LTkBb_j1Zmy4HI2eItaNhFXy-LIWF2hX88ijAQkYas0SrBvLC7d-SC58mLuvqwW4ZHntTCkIvRv1ZUs8fQPFCLl-WbmHcHhgavmVeEGXuCNAmzcAkslJ_eBghQUlbPPOqEAzNMgWrJkAzVcQBTxhYi2P8mQ4_c91L7mFeLe_NspFLCoWLnlIyfxwfVma6fpOGpGgg3eWydiOWT-WcNzNnY6P1TrN0gIL4QZvaD7VdBkvMJVtUeFHhxjZE5535Gddg1g1N469MesuGcIEISjOtjoBMpYMjmbRvIV4UyKSTFzSiqxZxOI0uSn1w-acf5g6G4tBmbfU_kSz1ass8ZV0rmiSrnTniNGWEg635jKCcTFND4KYuUJ1T21RlRP3q3btvejbzx29uuJxwWkR5KX0QR7CvVNQ; HSID=A2inQ1hxY_Akxsjaz; SSID=A9udoa1_RoYRXqu3b; APISID=LiX-ftG4pPZq-K4a/ALRoyKU_0PfrlgD5X; SAPISID=iVLeNwxPVKei4Q0q/Ayrp0XHMIQ5iq33wu; S=quotestreamer=6kIdj_SK-1Tobo0ctY2M3Q; NID=67=uQYk0bBnV8JbGB3UkfzG7f_1_vzYbCxH5BYU7_LjzwvN7WwEqe38sg93Um1eUxMlKDfgvuG2jpjL9UyjBI4gdGZDnMTDD21th6S0ptMJRh7HVCqxqzVMZMWtzQqaFOcCmHfNMp3wJBNwoynu2bsrJtNy5Kfz0TvaMkm-T6-_7h9PCTds7yzia46XVioSQl-8JYBbePrm9iaYYArRxMPl9mYs_5sEv5Noo4KRtKxUM4VpMGRx3Fykp1PMvMIU2T9cl2w9JiOGgryDuK-3b41Ihx326qpAdlAkB897bCf2TPl1AezTWf1UQOzTzv8XdPsvzd2sa4Sx0hND_Pi7rwPLHgzVOg9Sh3cqtUkjXycsKYWrVFIKfQCyy4g8sjKqwgvPQA; S=quotestreamer=6kIdj_SK-1Tobo0ctY2M3Q:talkgadget=aHfB4TSaCOCYGPzGrmtTLA
DNT: 1
Referer: https://579.talkgadget.google.com/talkgadget/d?token=AHRlWrqRUYIuQjAi1pH4ZFQhJWe92QrASI-PuFy3th_zCVENP71Zh90Pzs9Ch2JvJAf2qFw21V5oopm3dZRnzTTLl8UBAJ4co-b4WztC8owNbJhlYoVy-dS2lnsdBlXzcVph10ii8uFf&xpc=%7B%22cn%22%3A%22dV1z5gMTwj%22%2C%22tp%22%3Anull%2C%22osh%22%3Anull%2C%22ppu%22%3A%22https%3A%2F%2Fmightytext.net%2F_ah%2Fchannel%2Fxpc_blank%22%2C%22lpu%22%3A%22https%3A%2F%2F579.talkgadget.google.com%2Ftalkgadget%2Fxpc_blank%22%7D
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
X-Client-Data: CJa2yQEIprbJAQiptskBCN6WygE=
HTTP/1.1 200 OK
Age: 1
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Date: Wed, 15 Oct 2014 12:08:23 GMT
Server: ATS
Transfer-Encoding: chunked
Vary: X-Ssl
Via: http/1.1 media-border67.global.media.bf1.yahoo.com (ApacheTrafficServer [cMsSf ]), http/1.1 r18.ycpi.dcb.yahoo.net (ApacheTrafficServer [cMsSf ])
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Related
I need your help in putting together a post request.
The output I get is html, but the plan was to get the following:
Below are all the data for the desired item:
General
Request URL: https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml
Request Method: POST
Status Code: 200
Remote Address: 104.17.64.19:443
Referrer Policy: strict-origin-when-cross-origin
Response Headers
cache-control: no-cache
cf-cache-status: DYNAMIC
cf-ray: 76800ae95afc35b3-DME
content-encoding: br
content-type: application/json; charset=utf-8
date: Thu, 10 Nov 2022 16:07:42 GMT
expires: -1
pragma: no-cache
server: cloudflare
set-cookie: server_persistent=!zk3OrErnBetHZkiKJcby5Il79pzHsf7dxKD0PcVuB54Z2dznuEbqgGAVDWLDvoqpVSDnVq+Jtf91LHo=; path=/; Httponly; Secure
x-newrelic-app-data: PxQFUFRTDQMHR1NRBQkOVVABDhFORDQHUjZKA1ZLVVFHDFYPHjZWADdTRRcPAF0cXgMWAFJFaAcXQU4cBRAlEFEPXSpMVVgQH1UXUR1RHVBUAA9QVloUHgFIQ1YCAg9fAAgFAFZXUFYDUQBAFF5VXkAAZA==
Request Headers
:authority: dgslivebetting.betonline.ag
:method: POST
:path: /ngwbet.aspx/gvFrameHtml
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
content-length: 12
content-type: application/json; charset=UTF-8
cookie: \_xpid=574830729; \_xpkey=K_F3GRHECOTdjT306mOafHByLTxopGhY; LPVID=MxZmQyM2Q5OTFlOTU0ZTJk; \_hjSessionUser_2115245=eyJpZCI6IjQ3MzAxYmQwLTQ4ODgtNWNjMC1hZGZjLWJlZDBmNDgwZDJjZCIsImNyZWF0ZWQiOjE2NjY1NTY0MjQwOTIsImV4aXN0aW5nIjp0cnVlfQ==; CT.CONTENT.NA.STATUS=1; \_gid=GA1.2.1666042031.1667883501; PreviousUrlNav=%2Fsportsbook%2Flive-betting; chQuickBet=undefined; inputAmount=100.00; kameleoonVisitorCode=\_js_ti27yqxpj7dd4k1x; DD-LINK-NAREDIRECT=0; ASP.NET_SessionId=5acflzzgqtjdvsnjc5wtwuys; tz=Eastern%20Standard%20Time; btpdb.1PR3l09.dGZjLjY2ODI2ODU=U0VTU0lPTg; oddsfmt=dec; \_hjSession_2115245=eyJpZCI6Ijk2NzBiMjNkLWY4MGQtNDM5OS1hYWNhLWQyODBjNmZlYzNkMSIsImNyZWF0ZWQiOjE2NjgwOTM2NzY4OTUsImluU2FtcGxlIjpmYWxzZX0=; \_hjAbsoluteSessionInProgress=0; \_hjIncludedInSessionSample=0; LPSID-90263191=bLgFHbiuTjOcwCg1FgR16g; \__cf_bm=5LozQOf4P4COCn1rVD5emsVzukFSNbWdS7kvBVodzJ4-1668096251-0-AQ+nY5HeihIwV+gAI1oaFKJJxOtgXWs5czIr198Ffrh18P1q4nriEcszp/j7dwjuDjVuki1jlT6IByy2ewOCcXSUWavF+3MCcBF4Yb8sfDPVkvoSufxJ46feYuPiCiPcw0eW9oTUnrmZNcEkZ1732RDx6LWq1OElUvT0Uk6sk1n1; \_gat_UA-190679354-1=1; \_ga_KC6V6402HY=GS1.1.1668096234.18.1.1668096460.0.0.0; \_ga=GA1.1.1142263304.1666556424; server_persistent=!Tdbrpsz3tJ8jlNmKJcby5Il79pzHsfLVz91fFnDrXObiJE45d6idCUAVcW4Qmd/g598vNFaqTVuVRvk=
origin: https://dgslivebetting.betonline.ag
referer: https://dgslivebetting.betonline.ag/ngwbet.aspx
sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
x-newrelic-id: VgcFUVNTDxACV1NaDgIDVlw=
x-requested-with: XMLHttpRequest
Please help me figure out how I can get what I want.
My code:
import requests
import cloudscraper
scraper = cloudscraper.create_scraper()
url = 'https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml'
data = {"gameID":0}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
'Referer': "https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml"
}
r = requests.post(url, data=data, headers=headers)
print(r.text)
In order to get JSON back, you need to add the Content-Type header to your request.
Your current examples shows you are only sending these headers:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
'Referer': "https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml"
}
At the very least, you'll need to add Content-Type: application/json; charset=UTF-8 to the request, otherwise, requests is doing an application/x-www-form-urlencoded form post which is why you're getting back HTML from this site instead of JSON.
I'm working on a web scraper build in python. Until now I build the following code:
import requests
headers = {
'authority': 'truegamedata.com',
'accept': '*/*',
'x-requested-with': 'XMLHttpRequest',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.152 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'sec-gpc': '1',
'origin': 'https://truegamedata.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://truegamedata.com/weapon_builder.php',
'accept-language': 'pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7',
}
data = {
'weapon_name': '^%^5B^%^22Kilo 141^%^22^%^2C^%^22wz^%^22^%^5D'
}
response = requests.post('https://truegamedata.com/SQL_calls/base_data.php', headers=headers, data=data)
print(response.text)
For some reason, I get the following error:
<br />
<b>Fatal error</b>: Uncaught Error: Call to a member function execute() on bool in /home/customer/www/truegamedata.com/public_html/SQL_calls/base_data.php:29
Stack trace:
#0 {main}
thrown in <b>/home/customer/www/truegamedata.com/public_html/SQL_calls/base_data.php</b> on line <b>29</b><br />
Does anyone know why this is happening? And how I can get this response?
Here is the request from Chorme Dev tools:
Request URL: https://truegamedata.com/SQL_calls/base_data.php
Request Method: POST
Status Code: 200
Remote Address: 127.0.0.1:61696
Referrer Policy: strict-origin-when-cross-origin
cache-control: no-store, no-cache, must-revalidate
content-encoding: br
content-type: text/html; charset=UTF-8
date: Fri, 12 Feb 2021 20:08:45 GMT
expires: Thu, 19 Nov 1981 08:52:00 GMT
host-header: 8441280b0c35cbc1147f8ba998a563a7
pragma: no-cache
server: nginx
vary: Accept-Encoding
x-httpd-modphp: 1
x-proxy-cache-info: DT:1
:authority: truegamedata.com
:method: POST
:path: /SQL_calls/base_data.php
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: pt-BR,pt;q=0.9
content-length: 42
content-type: application/x-www-form-urlencoded; charset=UTF-8
cookie: PHPSESSID=375e8ebdfa9174d6db5eb8c1cda4411b; game=wz
origin: https://truegamedata.com
referer: https://truegamedata.com/weapon_builder.php
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
sec-gpc: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.152 Safari/537.36
x-requested-with: XMLHttpRequest
weapon_name: ["FR 5.56","wz"]
I tried to give as much information as possible, if anything is missing let me know
I am currently trying to get data off http://www.spotrac.com/ that requires being signed in. My current attempt uses this code(which I got by going through a bunch of other stack overflow questions on a similar topic)
from bs4 import BeautifulSoup as bs
from requests import session
payload = {
'id': 'contactForm',
'cmd': 'http://www.spotrac.com/signin/submit/',
'email': '*****',
'password': '*****'
}
with session() as c:
r_login = c.post('http://www.spotrac.com/signin/', data=payload)
print(r_login.headers)
response = c.get('http://www.spotrac.com/nba/cleveland-cavaliers/lebron-james')
print(response.cookies)
soup=bs(response.text, 'html.parser')
with open('ex.html','w') as f:
f.write(soup.prettify())
My current code does everything right, except I am not logged in when I'm making the request.
Thanks
You're sending POST request to a wrong URL, and with an incorrect payload as well.
POST http://www.spotrac.com/signin/submit/ HTTP/1.1
Host: www.spotrac.com
Connection: keep-alive
Content-Length: 86
Cache-Control: max-age=0
Origin: http://www.spotrac.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://www.spotrac.com/signin/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: cisession=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2206021e191bdbbaf955f111f67b961056%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A11%3A%22119.9.105.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A108%3A%22Mozilla%2F5.0+%28Windows+NT+6.1%3B+WOW64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F55.0.2883.87+Safari%2F537.36%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1485487245%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7Dd6089620b21ecce6837161605055ae04; _ga=GA1.2.910256341.1481865346; _gali=contactForm
redirect=http%3A%2F%2Fwww.spotrac.com%2F&email=sdfs%40gmail.com&password=lkasjdflksjad
HTTP/1.1 302 Found
Server: nginx
Date: Fri, 27 Jan 2017 04:21:16 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive
Set-Cookie: cisession=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22badb1275aee1cdad6736a6b4bb1ce809%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A11%3A%22119.9.105.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A108%3A%22Mozilla%2F5.0+%28Windows+NT+6.1%3B+WOW64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F55.0.2883.87+Safari%2F537.36%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1485490876%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7Dad486866c32cac526487707cea85b8a9; expires=Fri, 10-Feb-2017 04:21:16 GMT; path=/
Location: http://www.spotrac.com/register/
X-Powered-By: PleskLin
MS-Author-Via: DAV
As you can see from above session, the correct url should be http://www.spotrac.com/signin/submit/, and payload string is redirect=http%3A%2F%2Fwww.spotrac.com%2F&email=sdfs%40gmail.com&password=lkasjdflksjad, which is basically:
payload = {'redirect': 'http://www.spotrac.com/',
'email': mail_address,
'password': password}
Also make sure simulate headers with correct parameters, then you're good to go.
Guys i m scraping this url 'http://goldfilmesonline.com/jack-reacher-sem-retorno-legendado-online/' to try to get 1 link,but they have 2 that work for me, what happend to the site, i think it redirect, whem u play the video, 1 thing i know it send 2 media links whem i play the movie, those are the links i want to get because they work in VLC or Kodi so my firt code is to get the embeded url and it work fine
import requests
from bs4 import BeautifulSoup
a = requests.get('http://goldfilmesonline.com/jack-reacher-sem-retorno-legendado-online/')
soup = BeautifulSoup(a.content, 'html')
links = soup.find_all('iframe')
for i in links:
x = (i['src'])
if 'openload' in x:
print x
this is the result:
https://openload.co/embed/BQgJDIUtZ_w/
here is what i dont know what to do, i use fiddler and could get the headers request and response but i dont know what to parsed or params to give the so i took x and try to log this url and get the links i want but dindnt work here is what i try
import requests
url = 'https://openload.co/embed/BQgJDIUtZ_w/'
data = {'mime':'true'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'}
#response = requests.post(url,params=data,headers=headers)
#b = response.status_code
#x = response.text
#c = requests.get('https://openload.co/embed/BQgJDIUtZ_w',params=data,headers=headers)
#d = response.headers['Location']
#print d
headers2= {'Transfer-Encoding': 'chunked', 'Set-Cookie': '__cfduid=d11e26a118392f7f08c5df1e88e15b3f71479421090; expires=Fri, 17-Nov-17 22:18:10 GMT; path=/; domain=.openload.co; HttpOnly, _csrf=bf05bbd254877ef6c354a8fb3d4001938ce56f8141ab4704460eb96f946f790ca%3A2%3A%7Bi%3A0%3Bs%3A5%3A%22_csrf%22%3Bi%3A1%3Bs%3A32%3A%22QWePPbK2IdPoeI3nfzzW0cf1Xlu2xgVv%22%3B%7D; path=/; HttpOnly, _olbknd=w6; path=/', 'Server': 'cloudflare-nginx', 'Connection': 'keep-alive',
'Cache-Control': 'private', 'Date': 'Thu, 17 Nov 2016 22:18:10 GMT', 'CF-RAY': '30368e96ed2a5a56-BOS', 'Content-Type': 'text/html; charset=UTF-8'}
f = requests.post(url,params=data,headers=headers)
print f
this last code i get a response 400, but it should be 302 found
here is the fiddler results:
this is the request headers
GET /stream/BQgJDIUtZ_w~1479434968~73.248.0.0~BduDRWUg?mime=true HTTP/1.1
Host: openload.co
Connection: keep-alive
Accept-Encoding: identity;q=1, *;q=0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
Accept: */*
Referer: https://openload.co/embed/BQgJDIUtZ_w/
Accept-Language: en-US,en;q=0.8,pt-BR;q=0.6,pt;q=0.4
Range: bytes=0-
This is the Response headers
HTTP/1.1 302 Found
Date: Thu, 17 Nov 2016 02:09:42 GMT
Content-Type: video/mp4
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d9ff8e31a73be207f66c4717480c6afe41479348582;expires=Fri, 17-Nov-17 02:09:42 GMT; path=/; domain=.openload.co; HttpOnly
Cache-Control: private
Access-Control-Allow-Origin: *
Location:https://1j8b54.oloadcdn.net/dl/l/zHA2Jp0IKTcUySV8/BQgJDIUtZ_w/Jack.Reacher.Never.Go.Back.HDTS.XviD-TOM.avi.mp4?mime=true
Set-Cookie: _olbknd=w5; path=/
Server: cloudflare-nginx
CF-RAY: 302fa46268585a68-BOS
I'm using python 2.7 with requests and beautifulSoup to get Python to print me 1 or both this 2 links, first is the one in Request headers:
GET /stream/BQgJDIUtZ_w~1479434968~73.248.0.0~BduDRWUg?mime=true
the second is the one in response headers:
Location: https://1j8b54.oloadcdn.net/dl/l/zHA2Jp0IKTcUySV8/BQgJDIUtZ_w/Jack.Reacher.Never.Go.Back.HDTS.XviD-TOM.avi.mp4?mime=true
can someone help me
Here is what my XHR data looks like when captured in chrome
Request Header
POST my_url?X-Progress-ID=ee821652321919bc7ae61fbe0b625990&userpkgname=file_name.tar.gz HTTP/1.1
Host: 10.110.134.28
Connection: keep-alive
Content-Length: 17461
Accept: application/json, text/plain, */*
Origin: https://10.110.134.28
X-XSRF-TOKEN: bGIwdfFE-oaL_1yVrCzw0iHvv4yUHLC28xjw
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarybfK9jSdLoc2Mpj0i
Referer: https://10.110.134.28/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: XSRF-TOKEN=bGIwdfFE-oaL_1yVrCzw0iHvv4yUHLC28xjw; sid=s%3AXglfJNLQ9zzp3eHjQ2QOpk19kFKDDMvJ.ZMKyZd1Gx13lz2MnJgty5WncnilySzfoThGktkhlk4w
Payload
------WebKitFormBoundarybfK9jSdLoc2Mpj0i
Content-Disposition: form-data; name="package"; filename="file_name.tar.gz"
Content-Type: application/x-gzip
------WebKitFormBoundarybfK9jSdLoc2Mpj0i--
And this is how I am building my request.
files = {'package': (<file_name>, open(config_path, 'rb'), 'application/x-gzip')}
request.post(url, files=files)
This is how my request header looks like
{
'Content-Length' : '17449',
'Accept-Encoding' : 'gzip, deflate',
'Accept' : '*/*',
'User-Agent' : 'python-requests/2.10.0',
'Connection' : 'keep-alive',
'Cookie' : 'XSRF-TOKEN=zmklLEL0-gDJOfNBk113MuTpBkLo0j6MAzw0; sid=s%3AC2JZDCfpg_CgkU7qSlS5YTvWXwpgMX35.5nU7W02TPNYtMkIQ4W%2B1bjd87A7KyJbh3shoNqqADXE',
'Content-Type' : 'multipart/form-data; boundary=270d9e02bf214dc7a09c3081cba5b0e0',
'XSRF-TOKEN' : 'zmklLEL0-gDJOfNBk113MuTpBkLo0j6MAzw0'
}
When I make the request I get 502 bad gateway response that too after few seconds while on chrome I get 200 OK instantly
So most probably I am not building my request correctly. Any suggestions?