I am trying to do the same request from nodejs.
The python code is
import requests
r = requests.post(url,
data=data,
headers={
'User-Agent': self.ua,
'Content-Type': 'application/x-www-form-urlencoded'
}
)
and in node I tried node-fetch and Axios and request but not getting the same response, I also tried CURL from bash but getting the same response of node, I tried to print python headers print(r.request.headers) and copy paste it in node but getting different response
Axios.post(url, {
data,
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
"Content-Type": "application/x-www-form-urlencoded",
}
})
.then(text => console.log(text.data))
.catch(err => {
console.log(err);
});
I am getting different result in python I get what I expect but in node I am getting html response
Sorry, could not complete request because: <div class="tk-intro" style="font-size: 14px;color:#ff090f;">application information was not supplied.</div>
but in python works fine
I tried to print request headers and url and data and found I should to to convert data to query string like that
"appleId=email#gmail.com&accountPassword=xxxxxx"
instead of passing it as JSON
{
"appleID": "email#gmail.com",
"accountPassword": "xxxx"
}
Related
I am trying to run the following code for googleautoreply but it is giving the error 'Invalid JSON payload received.
Here is the Code:
import requests
import json
data = {"comment":"Thank you"}
headers =
{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/94.0.4606.81
Safari/537.36 Edg/94.0.992.50",
"Authorization": "Bearer {token},
} endpoint="https://mybusiness.googleapis.com/v4/accounts/{accounts_id}/locations/{location_id}/reviews/{reply_id}/reply"
var=requests.put(endpoint, data=data, headers=headers).json()
print(var)
After doing research I got to know that we have to pass the data in this way
data = """{"comment":"Thank you"}"""
I'm doing a GET request on POSTMAN. The request is just this URL https://www.google.com/search?q=pip+google-images&tbm=isch and I get a 853639 characters long response(Which is the response I want).
So I want to do the same thing with Python. I used the Postman's GENERATE CODE SNIPPETS and copied the code for Python Requests, I pasted it into my own python script and ran it.
But the response I got was only 22490 characters long(The response I don't want).
Why is it happening?
Python's code:
import requests
url = "https://www.google.com/search"
querystring = {"q":"pip google-images","tbm":"isch"}
headers = {
'cache-control': "no-cache",
'postman-token': "6b5e997f-6651-2178-1371-5d6a555984a7"
}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.text)
You need to set the User-Agent, e.g.
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
}
Include this in your headers.
I try to send form data to the following url: https://peepoo.gq
I have tried adding headers, and almost everything in the chrome network log that a normal browser sends, but I am always left with error 405. My code:
import requests
url = 'https://peepoo.gq/'
payload = {'content' : 'bruh'}
headers = {
'accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'user-agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
'content-type' : 'application/x-www-form-urlencoded',
'accept-encoding' : 'gzip, deflate, br',
'accept-language' : 'en-US,en;q=0.9',
'cache-control' : 'max-age=0',
'content-length' : '12',
'dnt' : '1'
}
params = {'guestbook_name' : 'main'}
r = requests.post(url, data=payload, headers=headers, params=params, allow_redirects=False)
print(r)
The following code is working for me:
import requests
url = 'https://peepoo.gq/sign'
payload = {'content' : 'pickle rick'}
headers = {
'content-type' : 'application/x-www-form-urlencoded',
}
params = {'guestbook_name' : 'main'}
r = requests.post(url, data=payload, headers=headers, params=params, allow_redirects=True)
basically, you need to allow redirects and your url needs to point to https://peepoo.gq/sign
You are using a POST method what is not accepted (said already by Klaus D.) and using the data parameter, what passes the information to the POST method also.
Apart from that you are passing twice the following key value pair: 'guestbook_name' : 'main'
Once in the url and once in the params.
I got 405 HTTP error, because of the default User-Agent: python-requests/2.27.1 header that requests module sends . Changing User-Agent header helped me:
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36'
}
url = 'https://example.com/some-path'
data = {'a': 1}
r = requests.post(url, data=data, headers=headers, allow_redirects=True)
print(r.status_code)
I'm using Python to collect prices data from southwest.com.
Trying to collect their Low Fare Calendar prices.
Example page: https://www.southwest.com/air/low-fare-calendar/select-dates.html?adultPassengersCount=1¤cyCode=USD&departureDate=2019-04-25&destinationAirportCode=MCO&originationAirportCode=MDW&passengerType=ADULT&returnAirportCode=&returnDate=&tripType=oneway
The needed data comes out from post request to https://www.southwest.com/api/air-booking/v1/air-booking/page/air/low-fare-calendar/select-dates.
The problem is in headers which starts with x-:
x-5ku220jw-a: I_ftfpDjDyifDufjizDcDs7teKDhlBY7UN6xgaX7hf8wp7=ZfV=ZiagmgMX1lgT1Yf8tIziO2PrYhLk6DprKAQChh-kEK2=QDyutU3E-gz6wgBpYQrUf8KlMjN6c0aP_95jAIV9BYr35sb66QM06CZP3IZrgF1XwfgNR0akQIc5tIaOIYagjTf1yAyP3louw57s3iZ_m07huHuaxFJI7lU_IFSvNg91IMb8riZ_RIcUrp7rNlpUffIUfqa8aIrEPkaSxiI=wPSP3k9=c1yvM8ikxhZ6h9t1kjSgmibYJInPU8hPfp-3uezOfeKLZIiQypKSQl14WSN_BPJURUaP1qa9R87EBFb8AqSv-ia7IiB0jlzSZ5IktfFfu2asJBII5lh5ss-XagNiZflSxYf8ZPVCljvSzl36bldUMFhj18zkI07=tgkXuU0UcZ7BPpb8tI3=3jr8-I-5QpfLBKWk3iq0bpK7KIsh7C3Ke1cB7t_S32kXhi_fBYN6VpIfu0-kyi753uzv0I-Q5=UsBYNP3DskijalfqWYP2kf5ly6mDILRqLimi7=AE3=tDcsM8-06fr7zDaLcc9I72zBRfIrLqN6Ch-b8D_f7fNuM__MCgaJLDkfhOBSwk-8yI-lusLPFvrN4fejQR79fsLmxFhrRS7jeEbItf_gBPKYS93w6YNimDrfaXR4uICfPKNP6fZP6ldEpciwKqNu7qMBRm3=mDavQ2szxqr=Cf9Yti-YuEU9c3I7t5WkQDs=LlDID2zSMp-gZI-BSf-0bkfURp3Y5fiRuirrB0aFBqf=SNVjLmBY5jakHlzURuu8cTM5wXKv3gaitlD8M3-vMIcYufiDrE4_fi-RhivW51bi7hY7jkhrhfbUBFh=6IaUbXhWTiu8R0Z_T37SQqW=jMLfA5sEBqrDARzQTqU=Nfkqyia7RTLrMUUgCFJbR2TIhl80I9BSLfaijYM6M5XYrCzvjIN6mU_j3jM-sRNiufLPTjUPm8K16Ua5tfr=b3=pfBacjF46fX7uMCujTsW=j0LS4lK8yWTIKlzQMYNhjfqF8WaPcIbIOsSIyp=8cqy6fvKS5de_Bg7v_IzhtYMf7FI8bluf7kJkZ8UFfqifSiSXBlc7ukXDkFJ=LiKs5pZP_f7jBFzvy5y_rRJ6_qM6Nd7PnL46xItbRu3PMUOEjIbSwgKtsHc9MFz9RI-BepBaLqi7A5zqZFapf0N6PYSiuFtj6iZ6LTsX3pbNSWL=w2USZpbPM3a=RI77fI-DnwiUMeSEcDjm0dyfZPb8Mi6jbE1folpkyf-vNMbUcUV9WfoXu5zEMrrj-pur7jr=MprvMYfjZdp5yqapmYrP3D4lteJ8ZphgMI9Y5W2WhjahM5rPRF-RfIILB2zXrDejnFzStjScMY7RurgpMIbsHDU=CIbYjI9ifsSxFfakif-9RU7sMgrmZ9XihWzkZPA8MI-9cR7sB8bPZYfY90asf5ch0davzIB7ulqrKu-8_PZubgL-ApbujDBfZjf8QDUrRjbfhpbThpJ=3iZ6_lkI1DUPsib8NR2pSfasVDw3uvLTqFBiuFbU7WU=jlD8Blbi7hu7MFq8hj30MmDR1jN6VDyStD4Q1jSP-FpMfgr7k2IjM0f83lkzklKhtFw7qlz=nI=ISqrS1g37flKPAizmAfS4ffvrqehI5fTD3fi=sI_YjtZ6Y2eVtj7=t0SNM9KURiakypa0_ib6-sKOcONujgKQ_dejhi7=4IcPQ5chjiNj3IXIrpbIhUjcZDrBfpas74JiuFV=rjf8UUH=3DsPckzkQIkgMuavhIZrZ21ycDpOag7vNLKXhq7PfDNPQCKuilKRulhSy3ij9DDIZI99cqi9fp7OcFJiB2zfjFAenj_PBq78MMI0ZDLSQ29mmDfrzhZPNFK1t93w3i9WZgIjQDeQBUq80qavQDevID-s7iy6Aj7STFJ63FsY5qTmUPO6MiVjsXBjMPfSmaLlbjaSnp7khFBYSg46ZqrbBT78tf1vmQa5yUyPwE-UR4S4=zN6_TZ669KB4fL=6iy6ZEBSmIa5miefM27LBqNPMDzSQqfS3UAUQhip7gs=322kt2ZPm8XPmixikacSCYMMxpBPQiZitiMjIfcEsp-9Rrz833LYus7k9IakRiVfYiQ0epu7ufLi9DaX5pa0UYiLRDd8jp-9SPsaUDIUR37Sm9ccrDd8_IrPV0JIZImiuTbsagaujULPs9hIjphgjYLi7eUSBezQb2zvjU3=mkzu1Ec=TqfI1m7s3gakhDz5AkUf7I-3BXL0hzKIupak_ffY5qSNfii0t3fj9kbUBL4uwF7kUi_8AjaXhiSgEFurL5svxI3_BF7v1dzPMDfjtTNujFq8Ty37tDc=MlU5Al-N6h7E883SM2I86Cc=r0AUR3I7ug7yRF1kygdzvP1XhgejrdL8Ah7=Ki-hujfirP_7tfuU=oWYfgaYIiaIMfJkifzszkU8PPMjADgpZpc9micIWp46BFM0flQ4Rse0tirZBU-lCWLm3YifrUJshfkXjY7P10_MjkaQMU4cukM_RI-S0lIyBXK5VEKchkZZRFd8TFQXuDZ_rezP3k_ZRlJEfT4utq7XBgf7jUfYmDSy-pabrIz9cFAFRFujhQSNR2ffjjNN=j3EBFBYZq7gwj7j0fL=MII=yhw6BkLY5gs=pgq8Z4mkmtevIjJUBIzk3qM6Q9trSY7uSF7=tIrEBUa8yhIrVYakAOQXSihj5DaR4FJEfU-7sDxg_rKpW9K0rlzvthSi7ptRRULj8WIjj8qURdPIRPT8losP3sLIfhLujp-bjb16bgWYS=A8mDbItmu8kjTISpuL7fqi55ZNcqrjwgavmq=gklkXuDsjti9lZ94D9I-hf9h8Qh3kQJasYqrQaqryRp373iZu7O-khX3YufUjQezmm3L9cFJs7gasuz9YjIZ_jisYugs6cf9kNUMr-DpjNqfjAfssuFAIAIcj3qMctFc7EqLNojTIhCxufBQEfSevQXhTxIz4RgL=eM_1Bh_8tiVM1Uhl7DIIJicELfILfdSXCjB8epcrIjr6mjPFDIsRZpbrBXKS68I7hkMfu3ejsq7kjIdTsCAe9i-vhscEToKJcfII7DdrTFzQL_4Tf8yf75J83JLDjEh8Qp3YjWpsA5zkoic=QFLIjq7E7Spjh8zSZXfEfTIYhjrjtgViKdLiu8swwq7SsNZ6P2Z6ZU3Yfkq8MPbrts9kRlpjLDI5AgUky4KPmfJk2q7=UCAm3DNgEFZ66I-==sqw18pkMfsbjp1zx8BjAMZ6iipfpDyvrJcsLjNNRh1vyi_rRqjkmfS4WSSPf3uaxFyPcjfsjiui3bhXjQWUID9uM61=Ajtjt2aDxFBPMta379s=iqLN=XKSMfushq75Ppu9t0aveDN6MFr1jlbUBqaOcIQkjIchhqf9RuVftLbDcYSptj4QLXh=ti6gA9bYhhauA9KQ382=UizLRIZNf8zRgiuncgLjyFX=iuNZ38s=tFZY7Ib8_DzS-DFDBpcvmSfI7iICaI_fhDsjU2y_LF38mInRKFuf7gIjcfbbBf7gQfBS_gMHvKuLRq9sujUD6lsPzF3SwI3YLh9IjIVj1fz6NFBDiFuI1iL=3qf4jleRtIN6yIN6j83BcIpj9IchAX1uu24XSsM=QiumNsJ-RfQkyI-7jCZu75kqxlhNzKcfhHs7ffN6MM9=jia5wu7pAi-FrIbFfgfgAWaQJfrI4gLrRpZfh6Z6ZgSv3iVFBWrlJIijRg=ZcFKkyYMvyoYFu24iuU15ME3EfY7Yr8z0QF-9D5VQbpKS6pNkpfa0QDrP0hfdx9tv0sKY75z06fLPiFJ8oYf5AIbsPU77ut7zuSa7t=3gEF3=qT7=61P=yrhsBFISmM-8xXKSpD1g6dMlff-RbDz_l2aI7I_9_XKvbi-1NfxjjWsrEfTPtL7LzjMffg1ufdrkEIb69FifjPh8AiwvMWsPc0agrDbbRoU7MhLksDy6x5N_fqLPhUu731_WLdrWdpK0aqaVhpruxlI83FcYYiIBEhP8SvUPmF-_Riue4t-ijSsv3qMfMiN_StL8m2aR5qrYtIx013a=tki83iIf57-kNI3SyYj6VXqLcFhS3jbFR3LjSiTUM3_S=Exj6f4u5FQP0kPky2LyRf98y3s0mvaNBPT8j5JEfeasjjrYujakb9BYwDdeujPfwFAmefIUM=3yIF-JRYaaxp_SL21vBF74BpMjrFh1tUS=z0T8bPQgtls=NDsa4l46N5Q86qfL_FLeVEQXrIXutRz1MphpAFTURUaTJg_gQpakcIuEAF1yBqa7f8yiticYuqNjSh7NBX46EE_lbgsER9JYt=hLRYNuWEJUR0auhjilth9ERU3=SX01w2_yRI6mMi_=QEp=ADzvLhq80gLWxidptsB96p-QVlIrokg1mI3tLEKkRfy6NIFpjDev3Y7rZlk=t2z9fsZ_lU3Yupxf5Hk8sl-g6YLItIaX7jSjtUt0niU=LjBEBBb030ujMsa9Rtr=o9XS3FXwwkL=mpx0yeJkjoJ=je-0csLfgFLgCIch58IfhiuaSfUrR9hDQFhS3Io3uYLy8Nyurp-Xm8Bf1fSvfpLEcDUfRldL3gSgcZyNzIqPZpHWf9JPygaS3IUPZ4Jcfx_fjfIjtIiYtaNP35u=wRJ=hWBYfRB=jiU8hvLFUOsj9Pzp9DIDTMa98YNPVDk8eiKeJlzSDs-w32UL-iz1Q59kQDBYMf4NVXv00jsB0iBjHWKh3ix66D-5AR9zbpJeJOvgtkZ_rDz=UtJ-sgaPSCQrfg9URf-srhNi7E3gQIuCjp7Px4K9Rf7YKq7LRFBSQ3gjSDz=3pZvQlXkZkq53jKupjWYtjaOBFZuBqakbDzPfpJI7UbrBlII7PQUZ93jyhf8tq39jXqIfP46LUBYjqifAXJsMfavTpKkVFtjMU-5zEaHxK6jVI-kCFJk6XMrMoSvm5hP9Qf=jPqj3ichIar=ZI-BYUMrM0a3tFxrbi-ktuh8QYa5Rg_bRpKQB2zXlhSjyFM6hIejYR3pMFK6mqim-DUNRYLFUi377pWIuIzkQkqYJXqrBhumME-v1=ZijCcst2JERieyJiaS_iIDVqSXjrKLoFZ696Mjmg2=_ib8rptj9LejZiI8bkMRtKK3rYf=wXJ=Bl4uJFsQRpK_cgLQ6kxffqzERD_D3qLItfKSAlU=V2ITxUK1k5IwiscpJfkp7qUERIL_cFXrRq7XBkz0siuMbhkvQUJixisEajISkp-OYjL_VdLIjq=IjhffA298jnqejsPhf27o4fLQk8RsuD9QQizpMHc=3iBY7pbEcIzDxMyu=p3j9FR1rgNFWqYcOlK66XWjii9EpI9s4lkIHpaX5Fq5Bk4PmqN65R4N4KN6Ak34fpcEM9tfugzPAI3Y7jTUoYTuht7zsIcLcdxf7F3BBXJ=6kIaNp_fJq7XKUSv3DMjmgNPZEhSTiLULgav=XKvjj7jZDzOI0NutX1uuhT8-D1ge8UD70LIDFa4fgSvyF986fA67H-ORiuDCMakQXq8bqffViUjQ94PxpN6t0akQkaJBi7P9kxru3=VaUzS39tjBKeft3a0SqLPjizRh91vQpZ_Ohb8tfhPfF9=3RaSefilRzaEhYavwkrjJYacjqffIqa9rpZNBDsSQqL8LwpjcDIjQiaPTif8tqaOMkXfu8K6329IBFX8BpAUBXKVufU=tFh8Mf3jcIyP4grS6FKiZIV81UbiWyz=3l9IficLjC16MFZNR93WsfzSjf7=fkSX3rK6Zf2=QUBE7RlgVh_=tkiWjYrPZFNih2e5tiLPKXJPMWsvykfiI3SvMiSjTe_Yjlpshgs7Bib=ZlUPtUKXIF-=_6vySKFwLIuj1s4fjiN6HDf8DeXpt9J1bIoW9KIzLt_E7MJIuIr=8qaFcV-kL2zQ6FzCKhLctgbP6q98tUrSQF-1VlKXrDLMfhasCXhS9WzOR63=VFzvxg7v1UL6Vf_PLqfjZdurTFt6fDzRhlcLLrcRrkFvZ
x-5ku220jw-b: 3k9rur
x-5ku220jw-c: A__57lBqAQAACgDSJrJqFbzAqmJVRyggwSLzliwMgtTT0dRA3TtiKWkLmSk8AawUHfD6K-8VrJMAAOfvAAAAAA==
x-5ku220jw-d: 0
x-5ku220jw-uniquestatekey: A-gE71BqAQAAbcLSHXYa-TmPShEFZzL7WRKJL2s2ZrEiksXMK8pm-YimkgmLAawUHfGuclcCwH8AABszAAAAAA==
I don't know from where they come and how can I get them but the code doesn't work without them(or one of them). I'm sure about that because I can get response when send requests to another URLs from the site where it doesn't need to use these headers.
So, the question is:
1. What these headers are?
2. What's the way(if it's possible) to get values for these headers?
My code:
import requests
data_url = 'https://www.southwest.com/api/air-booking/v1/air-booking/page/air/low-fare-calendar/select-dates'
payload = {
"adultPassengersCount": "1",
"currencyCode": "USD",
"departureDate": "2019-04-25",
"destinationAirportCode": "MCO",
"originationAirportCode": "MDW",
"passengerType": "ADULT",
"returnAirportCode": "",
"returnDate": "",
"tripType": "oneway",
"application": "air-low-fare-calendar",
"site": "southwest"
}
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
'x-api-key': 'l7xx944d175ea25f4b9c903a583ea82a1c4c'
}
r = requests.post(data_url, json=payload, headers=headers)
print(r.text)
I'm trying to automate the recovery of data from this website (The one I want is "
BVBG.086.01 PriceReport"). Checking with firefox, I found out that the request URL to which the POST is made is "http://www.bmf.com.br/arquivos1/lum-download_ipn.asp", and the parameters are:
hdnStatus: "ativo"
chkArquivoDownload_ativo "28"
txtDataDownload_ativo "09/02/2018"
imgSubmeter "Download"
txtDataDownload_externo_ativo [3]
0 "25/08/2017"
1 "25/08/2017"
2 "25/08/2017"
So, if I use hurl.it to make the request, the response is the correct 302 redirect (Pointing to a FTP URL where the requested files are, something like "Location: /FTP/Temp/10981738/Download.ex_"). (Example of the request here).
So I've tried doing the same with with the following code (Using python's library "requests", and I have tried both versions of request_body, trying to put it into the "data" parameter of the post method)
request_url = "http://www.bmf.com.br/arquivos1/lum-download_ipn.asp"
request_headers = {
"Host": "www.bmf.com.br",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"Referer": "http://www.bmf.com.br/arquivos1/lum-arquivos_ipn.asp?idioma=pt-BR&status=ativo",
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": "236",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1"
}
# request_body = "hdnStatus=ativo&chkArquivoDownload_ativo=28&txtDataDownload_ativo=09/02/2018&imgSubmeter=Download&txtDataDownload_externo_ativo=25/08/2017&txtDataDownload_externo_ativo=25/08/2017&txtDataDownload_externo_ativo=25/08/2017"
request_body = {
"hdnStatus" : "ativo",
"chkArquivoDownload_ativo": "28",
"txtDataDownload_ativo": "09/02/2018",
"imgSubmeter": "Download",
"txtDataDownload_externo_ativo": ["25/08/2017", "25/08/2017", "25/08/2017"]
}
result_query = post(request_url, request_body, headers=request_headers)
# result_query = post(request_url, data=request_body, headers=request_headers)
for red in result_query.history:
print(BeautifulSoup(red.content, "lxml"))
print()
print(result_query.url)
And what I get is the following response:
<html><head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found here.</body>
</html>
<html><head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found here.</body>
</html>
<html><head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found here.</body>
</html>
http://www.bmf.com.br/arquivos1/lum-arquivos_ipn.asp?idioma=pt-BR&status=ativo
And not the one I wanted (Which should point to the location of the file). What am I doing wrong here?