Im always getting the error message "Bad Request" when im trying to Post data to steam, i did lot of resears and i dont know how to fix this.
Post Values:
# Post Values
total = int(item['price'])
fee = int(item['fee'])
subtotal = total-fee
Cookies:
# Cookies
c = []
c.append('steamMachineAuthXXXXXXXXXXXXXXXXX='+steamMachineAuth)
c.append('steamLogin='+steamLogin)
c.append('steamLoginSecure='+steamLoginSecure)
c.append('sessionid='+sessionid)
cookie = ''
for value in c:
cookie = cookie+''+value+'; '
Headers:
# Headers
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4",
"Connection": "keep-alive",
"Host": "steamcommunity.com",
"Referer": hosturl+"market/listings/"+appid+"/"+item['market_hash_name'],
"Cookie": cookie,
"Origin": hosturl,
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36",
"X-Requested-With": "XMLHttpRequest"
}
Post data:
# Post Data
post = {
'sessionid': sessionid,
'currency': int(currency),
'subtotal': subtotal,
'fee': fee,
'total': total,
'quantity': 1
}
Url:
# url
url = hosturl+'market/buylisting/'+item['listingid']
Sending Request:
# Sending Request
se = requests.Session()
re = se.post(url, data=post, headers=headers)
print re.reason
Output:
Bad Request
I can't speak specifically about the Steam service as I haven't used it yet, but my experience with typical Bad Request responses is that you're either trying an HTTP verb that isn't supported or your request is not formatted correctly.
In your case, I suspect it's the latter.
My first candidate to look at is your cookie formatting. Are you sure you don't have characters that need to be escaped?
I could suggest using something like this instead:
c = {
'steamMachineAuthXXXXXXXXXXXXXXXXX': steamMachineAuth,
'steamLogin': steamLogin,
'steamLoginSecure': steamLoginSecure,
'sessionid': sessionid
}
cookie = '; ',join('{}="{}"'.format(k, v) for k, v in c.items())
Related
I found an Excel/VBA script here and am trying to translate it to Python. First is the VBA, then my Python attempt. How should I fix the Python code?
Public Function GetUPSDeliveryDate(ByVal id As String) As String
Dim body As String, json As Object
body = "{""Locale"":""en_US"",""TrackingNumber"":[""" & id & """]}"
With CreateObject("MSXML2.XMLHTTP")
.Open "POST", "https://www.ups.com/track/api/Track/GetStatus?loc=en_US", False
.setRequestHeader "Referer", "https://www.ups.com/track?loc=en_US&requester=ST/"
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "DNT", "1"
.setRequestHeader "Content-Type", "application/json"
.setRequestHeader "Accept", "application/json, text/plain, */*"
.send body
Set json = JsonConverter.ParseJson(.responseText)
End With
Excel VBA/JSON to scrape UPS tracking delivery
My attempt is:
def f(tn):
data = {"Locale":"en_US",
"TrackingNumber":f"[{tn}]",
"Referer": "https://www.ups.com/track?loc=en_US&requester=ST/",
"User-Agent": "Mozilla/5.0",
"DNT": "1",
"Content-Type": "application/json",
"Accept": "application/json, text/plain, */*"}
r = requests.post("https://www.ups.com/track/api/Track/GetStatus?loc=en_US", data=data)
print(r.json())
My guess is one problem is putting the body portion from VBA into data.
Right. Locale and TrackingNumber are part of the data. Everything else is a request header.
def f(tn):
data = {"Locale":"en_US",
"TrackingNumber":f"[{tn}]"}
headers = {
"Referer": "https://www.ups.com/track?loc=en_US&requester=ST/",
"User-Agent": "Mozilla/5.0",
"DNT": "1",
"Content-Type": "application/json",
"Accept": "application/json, text/plain, */*"}
r = requests.post("https://www.ups.com/track/api/Track/GetStatus?loc=en_US", data=data, headers=headers)
print(r.json())
I'm trying to populate json response issuing a post http requests with appropriate parameters from a webpage. When I run the script, I see that the script gets stuck and doesn't bring any result. It doesn't throw any error either. This is the site link. I chose three options from the three dropdowns from this form in that site before hitting Get times & tickets button.
I've tried with:
import requests
from bs4 import BeautifulSoup
url = 'https://www.thetrainline.com/'
link = 'https://www.thetrainline.com/api/journey-search/'
payload = {"passengers":[{"dateOfBirth":"1991-01-31"}],"isEurope":False,"cards":[],"transitDefinitions":[{"direction":"outward","origin":"1f06fc66ccd7ea92ae4b0a550e4ddfd1","destination":"7c25e933fd14386745a7f49423969308","journeyDate":{"type":"departAfter","time":"2021-02-11T22:45:00"}}],"type":"single","maximumJourneys":4,"includeRealtime":True,"applyFareDiscounts":True}
with requests.Session() as s:
s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
s.headers['content-type'] = 'application/json'
s.headers['accept'] = 'application/json'
r = s.post(link,json=payload)
print(r.status_code)
print(r.json())
How can I get json response issuing post requests with parameters from that site?
You are missing the required headers: x-version and referer. The referer header is referring to the search form and you can build it. Before journey-search you have to post an availability request.
import requests
from requests.models import PreparedRequest
headers = {
'authority': 'www.thetrainline.com',
'pragma': 'no-cache',
'cache-control': 'no-cache',
'x-version': '2.0.18186',
'dnt': '1',
'accept-language': 'en-GB',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/88.0.4324.96 Safari/537.36',
'content-type': 'application/json',
'accept': 'application/json',
'origin': 'https://www.thetrainline.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
}
with requests.Session() as s:
origin = "6e2242b3f38bbbd8d8124e1d84d319e1"
destination = "15bcf02bc44ea754837c8cf14569f608"
localDateTime = "2021-02-03T19:30:00"
dateOfBirth = "1991-02-03"
passenger_type = "single"
req = PreparedRequest()
url = "http://www.neo4j.com"
params = {
"origin": origin,
"destination": destination,
"outwardDate": localDateTime,
"outwardDateType": "departAfter",
"journeySearchType": passenger_type,
"passengers[]": dateOfBirth
}
req.prepare_url("https://www.thetrainline.com/book/results", params)
headers.update({"referer": req.url})
s.headers = headers
payload_availability = {
"origin": origin,
"destination": destination,
"outwardDefinition": {
"localDateTime": localDateTime,
"searchMethod": "DEPARTAFTER"
},
"passengerBirthDates": [{
"id": "PASSENGER-0",
"dateOfBirth": dateOfBirth
}],
"maximumNumberOfJourneys": 4,
"discountCards": []
}
r = s.post('https://www.thetrainline.com/api/coaches/availability', json=payload_availability)
r.raise_for_status()
payload_search = {
"passengers": [{"dateOfBirth": "1991-02-03"}],
"isEurope": False,
"cards": [],
"transitDefinitions": [{
"direction": "outward",
"origin": origin,
"destination": destination,
"journeyDate": {
"type": "departAfter",
"time": localDateTime}
}],
"type": passenger_type,
"maximumJourneys": 4,
"includeRealtime": True,
"applyFareDiscounts": True
}
r = s.post('https://www.thetrainline.com/api/journey-search/', json=payload_search)
r.raise_for_status()
print(r.json())
As Sers's reply, headers are missing.
When scrawling websites, you have to keep in mind anti-scrawling mechanism. The website will block your requests by taking into consideration your IP address, request headers, cookies, and various other factors.
Search url - http://aptaapps.apta.org/findapt/Default.aspx?UniqueKey=.
Need to get data for the zipcode(10017)
Sending post requests but I receive the search page(response from the search url) but not the page with results.
My code:
# -*- coding: UTF-8 -*-
import requests
from bs4 import BeautifulSoup, element
search_url = "http://aptaapps.apta.org/findapt/Default.aspx?UniqueKey="
session = requests.Session()
r = session.get(search_url)
post_page = BeautifulSoup(r.text, "lxml")
try:
target_value = post_page.find("input", id="__EVENTTARGET")["value"]
except TypeError:
target_value = ""
try:
arg_value = post_page.find("input", id="__EVENTARGUMENT")["value"]
except TypeError:
arg_value = ""
try:
state_value = post_page.find("input", id="__VIEWSTATE")["value"]
except TypeError:
state_value = ""
try:
generator_value = post_page.find("input", id="__VIEWSTATEGENERATOR")["value"]
except TypeError:
generator_value = ""
try:
validation_value = post_page.find("input", id="__EVENTVALIDATION")["value"]
except TypeError:
validation_value = ""
post_data = {
"__EVENTTARGET": target_value,
"__EVENTARGUMENT": arg_value,
"__VIEWSTATE": state_value,
"__VIEWSTATEGENERATOR": generator_value,
"__EVENTVALIDATION": validation_value,
"ctl00$SearchTerms2": "",
"ctl00$maincontent$txtZIP": "10017",
"ctl00$maincontent$txtCity": "",
"ctl00$maincontent$lstStateProvince": "",
"ctl00$maincontent$radDist": "1",
"ctl00$maincontent$btnSearch": "Find a Physical Therapist"
}
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4",
"Cache-Control": "max-age=0",
"Content-Length": "3025",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "aptaapps.apta.org",
"Origin": "http://aptaapps.apta.org",
"Proxy-Connection": "keep-alive",
"Referer": "http://aptaapps.apta.org/findapt/default.aspx?UniqueKey=",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
}
post_r = session.post(search_url, data=post_data, headers=headers)
print(post_r.text)
Short Answer:
try to replace:
post_r = session.post(search_url, data=post_data, headers=headers)
to:
post_r = session.post(search_url, json=post_data, headers=headers)
Long Answer:
For POST method, there are many kinds of data types to post in. Such as form-data, x-www-form-urlencoded, application/json, file and etc.
You should know what is the type of the post data. There is a brilliant chrome plugin called postman. You can use it to try different data type and find what is the correct one.
After you find, use the correct parameter key in requests.post, the parameter data if for form-data and x-www-form-urlencoded. The parameter json is for json format. You can reference the document of requests to know more about the parameter.
I'm trying to scrape WizzAir for personal use. Can't understand what's wrong with my code. Could it be incorrect payload object or cookies?
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36",
"Accept": "application/json, text/plain, */*",
"Accept-Encoding": "gzip, deflate, sdch, br",
"Accept-Language": "en-US,en;q=0.8,lt;q=0.6,ru;q=0.4",
"Origin": "https://wizzair.com",
"Referer": "https://wizzair.com/"
}
search_url = "https://wizzair.com/lt-LT/FlightSearch"
session = requests.Session()
r = session.get("https://be.wizzair.com/3.8.2/Api/asset/yellowRibbon", headers=headers, allow_redirects=False)
session_id = r.cookies["ASP.NET_SessionId"]
cookies = {
"ASP.NET_SessionId": session_id,
"HomePageSelector": "FlightSearch",
}
# wizz_url = "https://be.wizzair.com/3.8.2/Api/search/search"
wizz_url = "https://be.wizzair.com/3.8.2/Api/asset/farechart"
payload = {"flightList":[{"departureStation":"VNO","arrivalStation":"FCO","departureDate":"2017-02-20"}],"adultCount":1,"childCount":0,"infantCount":0,"wdc":True, "dayInterval":3}
r = session.post(url=wizz_url,data=payload,headers=headers, cookies=cookies)
print r.content
>>> {"validationCodes":["FlightCount_MustBe_OneOrTwo"]}
I run this - even without session and cookies - and get some data.
You have to send it as JSON - using json=payload
import requests
payload = {
"flightList":[
{
"departureStation": "VNO",
"arrivalStation": "FCO",
"departureDate": "2017-02-20"
}
],
"adultCount": 1,
"childCount": 0,
"infantCount": 0,
"wdc": True,
"dayInterval": 3
}
url = 'https://be.wizzair.com/3.8.2/Api/search/search'
r = requests.post(url, json=payload)
print(r.text)
data = r.json()
print(data['outboundFlights'][0]['flightNumber'])
If you will have to use cookies and headers then use Session and then you don't have to copy cookies and headers from one request to another.
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36",
#"Accept": "application/json, text/plain, */*",
#"Accept-Encoding": "gzip, deflate, sdch, br",
#"Accept-Language": "en-US,en;q=0.8,lt;q=0.6,ru;q=0.4",
}
s = requests.Session()
s.headers.update(headers)
# to get cookies
r = s.get("https://www.wizzair.com/")
payload = {
"flightList":[
{
"departureStation": "VNO",
"arrivalStation": "FCO",
"departureDate": "2017-02-20"
}
],
"adultCount": 1,
"childCount": 0,
"infantCount": 0,
"wdc": True,
"dayInterval": 3
}
url = 'https://be.wizzair.com/3.8.2/Api/search/search'
r = s.post(url, json=payload)
print(r.text)
data = r.json()
print(data['outboundFlights'][0]['flightNumber'])
import urllib.request
url = ""
http_header = {
"User-Agent": "Mozilla/5.0(compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0) like Gecko",
"Accept": "text/html, application/xhtml+xml, */*",
"Accept-Language": "ko-KR",
"Content-type": "application/x-www-form-urlencoded",
"Host": ""
}
params = {
'id': 'asdgasd',
'call_flag': 'idPoolChk',
'reg_type': 'NY'
}
data = urllib.parse.urlencode(params).encode()
req = urllib.request.Request(url, data)
response = urllib.request.urlopen(req)
the_page = response.read()
print(the_page)
When I run this program, I get:
\xec\x95\x84\xec\x9d\xb4\xeb\x94\x94\xea\xb0\x80 \xec\xa4\x91\xeb\xb3\xb5\xeb\x90\xa9\xeb\x8b\x88\xeb\x8b\xa4
How can I transform this to Korean?
It's UTF-8.
print(the_page.decode('utf-8'))
assuming your console will handle those characters.