HTTP requests.post fails - python

I'm using the python requests library to get and post http content. I have no problem using the get function but my post function seems to fail or not do anything at all. From my understanding the requests library the POST function automatically encodes the data you send but I'm not sure if that's actually happening
code:
data = 'hash='+hash+'&confirm=Continue+as+Free+User'
r = requests.post(url,data)
html = r.text
by checking the "value" of html I can tell that the return response is that of the url without the POST.

You're not taking advantage of how requests will encode it for you. To do so, you need to write your code this way:
data = {'hash': hash, 'confirm': 'Continue as Free User'}
r = requests.post(url, data)
html = r.text
I can not test this for you but this is how the encoding happens automatically.

post(url, data=None, **kwargs)
Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.

import requests
url = "http://computer-database.herokuapp.com/computers"
payload = "name=Hello11111122OKOK&introduced=1986-12-26&discontinued=2100-12-26&company=13"
headers = {
'accept-language': "en-US,en;q=0.9,kn;q=0.8",
'accept-encoding': "gzip, deflate",
'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
'content-type': "application/x-www-form-urlencoded",
'cache-control': "no-cache",
'postman-token': "3e5dabdc-149a-ff4c-a3db-398a7b52f9d5"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)

Related

Letterboxd api Bad Request (400) for no apparent reason python

I'm trying to access the Letterboxd api to get an authentification token but I keep getting Bad Request feedback. Here's my code:
import requests
import urllib
import json
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': 'application/json'
}
url = 'https://api.letterboxd.com/api/v0/auth/token'
body = {
'grant_type': 'password',
'username': 'myname',
'password': 'mypassword'
}
response = requests.post(
url+urllib.parse.urlencode(headers), data=json.dumps(body))
print('\n')
print(response.text)
Any idea what I did wrong? The response is just an HTTP document with no info besides the Error message "400 Bad Request". Heres the documentation if it helps https://api-docs.letterboxd.com/#auth
First: I can't test if all code works.
You send headers in url but you should send it as part of headers=.
And when you use module requests then you can use json=body instead of data=json.dumps(body)
response = requests.post(url, headers=headers, json=body)
But header 'Content-Type': 'application/x-www-form-urlencoded' suggests that you will send data as form data, not json data - which needs data=body without converting to json
response = requests.post(url, headers=headers, data=body)
EDIT:
In your link to documentation I see link to python module (in part "Example implementations") - so maybe you should use it instead of writing own code because documentation mentionts something about "All API requests must be signed" (in part "Request signing") - and this may need extra code to create it.

REST API post request results in successful request but not creation

I've been trying to make a Post Request to my CRM API. The CRM API is very vague:
"You can use POST request when you wish to create records. You POST a JSON encoded string to the servers and it will return a single instance of the record."
POST /{object_name}
Example:
Request URL (POST):
/accounts
Request Body (JSON):
{
"name": "testing API"
}
I've had plenty of success making GET requests regularly, but POST is not working out so easily.
url = "https://apiv4.reallysimplesystems.com/accounts?<KEY>"
payload = {"name":"Ziggy","owner":"XYZ","addresscounty/state":"Awe","source":"Space"}
headers = {
'Content-Type': 'application/json',
'Cookie': 'XSRF-TOKEN=<TOK>; really_simple_systems_session=<KEY>'
}
response = requests.post(url, headers=headers, data=payload)
I get a status code 200 when I run this, but I'm really looking for that 201. The only clue that I've got to follow at this point is that when I run:
response.json()
I get the error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I've tried switching the response parameters to json:
response = requests.post(url, headers=headers, json=payload)
I've tried ensuring that my payload is json by using json.dumps():
payload = {"name":"Ziggy","owner":"XYZ","addresscounty/state":"Awe","source":"Space"}
payload = json.dumps(payload)
And I've tried all sorts of other shenanigans that I can't even recall at this point. Does anyone have any idea where I'm going wrong here? The 200 status code makes me feel painfully close.
Replace <AUTH_TOKEN> with your auth token
url = "https://apiv4.reallysimplesystems.com/accounts/"
payload = {"name":"Ziggy"}
headers = {
'Content-Type': 'application/json',
'Authorization': 'Bearer <AUTH_TOKEN>',
}
response = requests.post(url, headers=headers, data=payload)
Problem solved:
import json
import requests
url = "https://apiv4.reallysimplesystems.com/accounts"
payload = {"name":"RSS Test Joe Bloggs","addresscounty/state":"Maryland","source":"Word of Mouth"}
payload = json.dumps(payload)
headers = {
'Content-Type': 'application/json',
'Authorization': 'Bearer <AUTH KEY>'
}
response = requests.post(url, headers=headers, data=payload)
Rather than using Postman's code which included the in the URL and used :
'Cookie': 'XSRF-TOKEN=<TOK>; really_simple_systems_session=<KEY>'
I replaced it with a standard Authorization header. Secondly, I found that using json.dumps(payload) and json = payload was resulting in a Bad Request.

HtmlResponse working in Scrapy Shell, but not in script?

I'm using scraperAPI.com to handle IP rotation for a scraping job I'm working on and I'm trying to implement their new post request method, But I keep receiving a 'HtmlResponse' object has no attribute 'dont_filter' error. Here is the custom start_requests function:
def start_requests(self):
S_API_KEY = {'key':'eifgvaiejfvbailefvbaiefvbialefgilabfva5465461654685312165465134654311'
}
url = "XXXXXXXXXXXXXX.com"
payload={}
headers = {
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'x-requested-with': 'XMLHttpRequest',
'Access-Control-Allow-Origin': '*',
'accept': 'application/json, text/javascript, */*; q=0.01',
'referer': 'XXXXXXXXXXX.com'
}
client = ScraperAPIClient(S_API_KEY['key'])
resp = client.post(url = url, body = payload, headers = headers)
yield HtmlResponse(resp.url, body = resp.text,encoding = 'utf-8')
The weird part is that when I execute this script piecewise in scrapy shell it works fine and returns the proper data, Any insight into this issue would be GREATLY appreciated? currently 4 hours into this problem.
Notes:
Client.post returns a response object
Not my real API key
client.post doesn't have a body method
The error you get is caused by returning the wrong type (a Response).
From the docs for start_requests:
This method must return an iterable with the first Requests to crawl for this spider.
It seems the easiest solution would be using a scrapy request (probably a FormRequest) to the API url, instead of using ScraperAPIClient.post().
You should be able to use ScraperAPIClient.scrapyGet() to generate the correct url, but I have not tested this.
If you would prefer to continue using the official api library, a slightly more complicated option is Writing your own downloader middleware.

HTTP headers - Requests - Python

I am trying to scrape a website in which the request headers are having some new (for me) attributes such as :authority, :method, :path, :scheme.
{':authority':'xxxx',':method':'GET',':path':'/xxxx',':scheme':'https','accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8','accept-encoding':'gzip, deflate, br','accept-language':'en-US,en;q=0.9','cache-control':'max-age=0',GOOGLE_ABUSE_EXEMPTION=ID=0d5af55f1ada3f1e:TM=1533116294:C=r:IP=182.71.238.62-:S=APGng0u2o9IqL5wljH2o67S5Hp3hNcYIpw;1P_JAR=2018-8-1-9', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/68.0.3440.84Safari/537.36', 'x-client-data': 'CJG2yQEIpbbJAQjEtskBCKmdygEI2J3KAQioo8oBCIKkygE=' }
I tried passing them as headers with http request but ended up with error as shown below.
ValueError: Invalid header name b':scheme'
Any help would be appreciated on understanding and guidance on using them in passing request.
EDIT:
code added
import requests
url = 'https://www.google.co.in/search?q=some+text'
headers = {':authority':'xxxx',':method':'GET',':path':'/xxxx',':scheme':'https','accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8','accept-encoding':'gzip, deflate, br','accept-language':'en-US,en;q=0.9','cache-control':'max-age=0','upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/68.0.3440.84Safari/537.36', 'x-client-data': 'CJG2yQEIpbbJAQjEtskBCKmdygEI2J3KAQioo8oBCIKkygE=' }
response = requests.get(url, headers=headers)
print(response.text)
Your error comes from here (python's source code)
Http headers cannot start with a semicolon as RFC states.
:authority, :method, :path, :scheme are not http headers
https://en.wikipedia.org/wiki/List_of_HTTP_header_fields
':method':'GET'
defines http request method
https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods
and
:authority, :path, :scheme
are parts of URI https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Generic_syntax

Python Requests Code 141 Error

I'm trying to use requests in python to post a json dictionary to a url. I need to get a string back from the url but I keep getting a code 141 error -{"code":141,"error":"Missing a github repository link"}. I'm using this website(http://docs.python-requests.org/en/latest/user/quickstart/) to do requests.
Any ideas on why I keep getting that error? Code is below.
import requests
import json
payload = { "email" : "jade#gmail.com", "github" : "https://github.com/"}
headers = {'content-type': 'application/json', "Accept": 'application/json'}
r = requests.post("http://challenge.code2040.org/api/register", params = payload, headers = headers)
print(r.url)
print r.text
Update: The suggestion worked but now I'm getting an{"code":141,"error":"success/error was not called"} error when I try to save the response I recieve from the url into a variable and then post it back to a different url.
#Store the token into a variable
token = r.text
payload = { "token" : token}
headers = {'content-type': 'application/json', "Accept": 'application/json'}
r = requests.post("http://challenge.code2040.org/api/getstring", json = payload, headers = headers)
print r.text
Since you are making a POST request and you need to provide a JSON in the request body, use json argument, not params:
r = requests.post("http://challenge.code2040.org/api/register",
json=payload,
headers=headers)
(tested - got back a token)
Note that json argument was introduced in requests 2.4.2.

Categories

Resources