I'm interested in writing clean, readable and maintainable code. Some time we have to interact with external API. For example httpbin.org. For me it would be a good idea to somehow fix request call result structure in code. For example httpbin.org/get returns:
{
"args": {},
"headers": {
"Accept": "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "ru,en;q=0.9",
"Host": "httpbin.org",
"Referer": "https://httpbin.org/",
"Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Yandex\";v=\"22\"",
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua-Platform": "\"macOS\"",
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-origin",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.167 YaBrowser/22.7.5.1018 Yowser/2.5 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-456c9715-53b4jhdsjhbjdh3dskjea0"
},
"origin": "134.51.34.46",
"url": "https://httpbin.org/get"
}
Now I see the way to convert the json into NamedTuple/dataclass/Pydantic.
I wonder to know if there is some common approaches or maybe ready solutions/libs or so for this? Also I need this for accessing through dot.
Related
I am getting this error when I use the edit question API:
{"error_id":400,"error_message":"site is required","error_name":"bad_parameter"}
This is my code:
r = requests.post(f"https://api.stackexchange.com/2.3/questions/{qid}/suggested-edit/add", json={"access_token": at, "site": "stackoverflow", "title":t,"body":str(b),"key":key})
From the official docs page I'm getting the javascript request structure as this
await fetch("https://api.stackexchange.com/2.3/questions/11/suggested-edit/add", {
"credentials": "include",
"headers": {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0",
"Accept": "*/*",
"Accept-Language": "en-US,en;q=0.5",
"Content-Type": "application/x-www-form-urlencoded",
"X-Requested-With": "XMLHttpRequest",
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-origin"
},
"referrer": "https://api.stackexchange.com/docs/create-question-suggested-edit",
"body": "id=11&body=111&key=111&access_token=111&preview=true&filter=default&site=stackoverflow",
"method": "POST",
"mode": "cors"
});
In Python you can use like this
r = requests.post(f"https://api.stackexchange.com/2.3/questions/{qid}/suggested-edit/add",headers={"Content-Type": "application/x-www-form-urlencoded"}, data={"access_token": at, "site": "stackoverflow", "title":t,"body":str(b),"key":key,"id":qid})
You are using the requests.post function incorrectly. If you'd like to send parameters with your request, use the params argument and pass the parameters as a dictionary.
r = requests.post(f"https://api.stackexchange.com/2.3/questions/{qid}/suggested-edit/add",params={"access_token": at, "site":"stackoverflow","title":t,"body":str(b),"key":key})
import requests
session = requests.Session()
data = "memberNo=&url=&cyberId={}&cyberPass={}"
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7",
"Cache-Control": "max-age=0",
"Connection": "keep-alive",
"Content-Length": "57",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "www.sonofelicecc.com",
"Origin": "https://www.sonofelicecc.com",
"Referer": "https://www.sonofelicecc.com/login.dp/dmparse.dm",
"sec-ch-ua": '" Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "Windows",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "same-origin",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36",
}
main_headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7",
"Cache-Control": "max-age=0",
"Connection": "keep-alive",
"Host": "www.sonofelicecc.com",
"Referer": "https://www.sonofelicecc.com/login.dp/dmparse.dm",
"sec-ch-ua": '" Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "Windows",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "same-origin",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36",
}
login_req = session.post("https://www.sonofelicecc.com/doLogin.dp/dmparse.dm", headers=headers, data=data)
main_resp = session.get("https://www.sonofelicecc.com/main.dp/dmparse.dm", headers=main_headers)
if main_resp.text.find(username) != -1:
print("login success")
I am trying to login with requests without selenium. I've first tried login with selenium; however it was too slow; thus, I saved the selenium's cookie and return it to requests for faster work.
Then, I want to make all working with only requests. I've loaded all the headers correctly, but it doesn't work at all. How can I?
when you try to display a get request in the text, it gives out incomprehensible characters, I would like it to produce normal text
Code:
headers = {
"authority": "www.ozon.ru",
"method": "GET",
"path": "/product/playstation-5-digital-edition-339866183/?asb=ZdbNZjh%252BgUCDpV0uw5ZLJUkaSn2wNH%252FSaAKJ%252BAxhX2M%253D&asb2=ayxVVx0ddcEtoLM3AwfnfVSDeZSpnVMgJu1dkk3rkjo&keywords=playstation+5&sh=0OBU25Oz",
"scheme": "https",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br",
"accept-language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7",
"cache-control": "max-age=0",
"cookie": "__Secure-ext_xcid=704b112d0788105d7206457724d88846; _gcl_au=1.1.954892941.1622892064; visid_incap_2251426=KU5k7V3nRbi5hIuCe4vdFn1su2AAAAAAQUIPAAAAAADSheykofd2VQeH1V6sdne6; visid_incap_1101384=ng7gdJKQTtaTXDhwXw0BERteu2AAAAAAQkIPAAAAAACAxLmcAby/fnf6k3vP36Lor9I9dWxx3mbg; __Secure-ab-group=30; _abck=AD1EC4FEF31E5162861A4C24D2CA7963~-1~YAAQVxndWMOJ1Bl6AQAAzuyMdwYmY8viAq7FQPAI0crJs+Y7Tol5pA9DDuINFgy8m+dW33GrxDi2sthCys8Q8xdFoZ5b/+cj885D7t6jQxlVTWRyFksPOyCfG+aPZcNjWLG4gtLYGhig4GmVY2IhbziLiACrJVZ9tvvQe+bPDscWtCGH5oFB2KDTmr+/5anJzP52dInIJRinf0G36Uv6LmTBvJ5oqmtHns+wdvWHV2/XtFBwUrKukPL/yB4I534FenLEKBs/go7uQS0q8XCAoeXQuHxE+XXEHteC3ViGCfdsi83AQgjjXemaeBg6rIcc6GOo4HS+NPR/o20jeZaNPOw21BZoSvhmSzk3WWAoOxqjayhWTKVE/Uu0k/2n3yS2XuFjUsw9nmMtOslXKyWPHYWcAvAFzw==~-1~-1~-1; _gcl_dc=GCL.1631556671.CjwKCAjwyvaJBhBpEiwA8d38vJ0cwc6gyFvDIalkCBIdbC18GmVXVD0XxIgwZoz--ClC6ErOz2uVmRoCw6oQAvD_BwE; visid_incap_1285159=FmeU/lKJQvaWmkgD2CjChIi8QGEAAAAAQUIPAAAAAADss4s71kpz7hK78LrQuFFa; __Secure-user-id=0; _gcl_aw=GCL.1633529190.CjwKCAjwkvWKBhB4EiwA-GHjFrQIvq2PTJYu-I5iwQo9hU06pvsUVvjj37nTH7ACBQDGNCG1NvNHlBoCIvEQAvD_BwE; nlbi_1101384=wNPJODVEXFvoYA/LK8plmQAAAACQSMQv10AIEBv9M1qG+ZgE; incap_ses_584_1101384=8gq4QeKdPVbcmBCIpMkaCMfCZWEAAAAAxQKn2KtmDQeSxEYbjPxM4Q==; xcid=b0d40e42f20927f1b6ef2f74056069fb; incap_ses_633_1101384=NoBfaWnlixsskJIVP97ICA2dZmEAAAAARs7zxjAA1ruartIDr0d2SA==; __Secure-access-token=3.0.BrCcd3kzR2aUcKDrHyfVvw.30.l8cMBQAAAABhQLuyDO5qoKN3ZWKgAICQoA..20211013104710.f77kLnpPyPCZUz33bipJ1qSFh7n4QIBACd22xU-M_sE; __Secure-refresh-token=3.0.BrCcd3kzR2aUcKDrHyfVvw.30.l8cMBQAAAABhQLuyDO5qoKN3ZWKgAICQoA..20211013104710.CAQWWNrTHcPBdzYVN9iOE7QB4LwfW4rmHjDJEszki5A; incap_ses_585_1101384=y3r7Vz0DllfLWjWKF1ceCA2dZmEAAAAAne5pnSZ7U0xGgsv7j+fBMA==",
"referer": "https://www.ozon.ru/cart",
"sec-ch-ua": '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "Windows",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "same-origin",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
}
r = requests.get('https://www.ozon.ru/product/playstation-5-digital-edition-339866183/?asb=ZdbNZjh%252BgUCDpV0uw5ZLJUkaSn2wNH%252FSaAKJ%252BAxhX2M%253D&asb2=ayxVVx0ddcEtoLM3AwfnfVSDeZSpnVMgJu1dkk3rkjo&keywords=playstation+5&sh=0OBU25Oz', headers=headers).text
print(r)
response
how to make normal text output?
It looks like you're getting compressed output. (You could verify that by printing r.headers and looking at Content-Encoding.)
Remove the
"accept-encoding": "gzip, deflate, br",
request header because that claims you can accept brotli-compressed content, which Requests by default doesn't handle.
Since you specified accept-encoding, the server performed compression before sending the data to you. ~~It's a Zip compression (begins with PK)~~
Just don't specify accept-encoding at all and the server will likely send you an uncompressed data stream.
More about accept-encoding
hi every one im newbie in programing.and i need help to work my code i need to post this in formation to this web site https://live.adyen.com/hpp/completeCard.shtml.but i get error pls help...
import requests
headers = {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image /apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
"Cache-Control": "max-age=0",
"Connection": "keep-alive",
"Content-Length": "5185",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "live.adyen.com",
"Origin": "https://live.adyen.com",
"Referer": "https://live.adyen.com/hpp/pay.shtml",
"sec-ch-ua-mobile": "?0",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "same-origin",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"}
payload = {"card.cardNumber": "4001 0200 0000 0009",
"card.cardHolderName": "Juan Tamad",
"card.expiryMonth": "03",
"card.expiryYear": "2030",
"card.cvcCode": "737"}
r = requests.post("https://live.adyen.com/hpp/completeCard.shtml/post", headers=headers, data=payload)
print(r.text)
i got this error
There was a problem displaying this page. Please try again later, or verify your input on the previous page
I am trying to extract information for all the jobs on this website:
https://www.americanmobile.com/travel-nursing-jobs/search/
Looking at the network activity tab, it looks like all the data I need comes from a POST request made here:
https://jobs.amnhealthcare.com/api/jobs//search. I have attached an image that may help confirm exactly what I am referencing.
example_1
I wrote the following code in Google Colab to try to at least get the first 10 results. Referencing python requests POST with header and parameters, I know a lot of the headers may not even be necessary. I have tried sending this request without any headers at all.
Is what I'm trying to do even possible? I have only gotten a 400 response code so far.
If it is possible to accomplish this, is it possible to extract this information for all 4k + jobs?
import requests
import re
payload = {
"PageNumber": "1",
"JobsPerPage": "10",
"Filters": {
"Locations": "[]",
"Skillset": "[]",
"StartDates": "[]",
"Shifts": "[]",
"Durations": "[]",
"IsCovid19Job": "false",
"Exclusive": "false",
"Skillsets": "[]",
"DivisionCompanyId": "2",
"LocationSearch": "",
"KeywordSearch": "",
"DaxtraJobIds": "[]"
},
"SortOrder": {
"Header": "MaxPayRate",
"SortDirection": "Descending"
}
}
headers = {
"Host": "jobs.amnhealthcare.com",
"Connection": "keep-alive",
"Content-Length": "315",
"Accept": "application/json, text/plain, */*",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36",
"Content-Type": "application/json;charset=UTF-8",
"Origin": "https://www.americanmobile.com",
"Sec-Fetch-Site": "cross-site",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Dest": "empty",
"Referer": "https://www.americanmobile.com/",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9"
}
response = requests.post('https://jobs.amnhealthcare.com/api/jobs//search' , data=p , headers=headers)
Thank you
The formatting of your data wasn't entirely correct. This should work:
import requests
headers = {
"Host": "jobs.amnhealthcare.com",
"Connection": "keep-alive",
"Content-Length": "315",
"Accept": "application/json, text/plain, */*",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36",
"Content-Type": "application/json;charset=UTF-8",
"Origin": "https://www.americanmobile.com",
"Sec-Fetch-Site": "cross-site",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Dest": "empty",
"Referer": "https://www.americanmobile.com/",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9"
}
data = '{"PageNumber":1,"JobsPerPage":10,"Filters":{"Locations":[],"Skillset":[],"StartDates":[],"Shifts":[],"Durations":[],"IsCovid19Job":false,"Exclusive":false,"Skillsets":[],"DivisionCompanyId":2,"LocationSearch":"","KeywordSearch":"","DaxtraJobIds":[]},"SortOrder":{"Header":"MaxPayRate","SortDirection":"Descending"}}'
response = requests.post('https://jobs.amnhealthcare.com/api/jobs//search', headers=headers, data=data)
You can now adapt JobsPerPage and PageNumber to retrieve all the posts you need in a for-loop.