I'm trying to interface with a RESTful API that accepts application/x-protobuf objects in post.
.proto example object:
message register
{
required string name = 1;
required int32 id = 2;
}
using requests I'm able to POST this message to the server and receive a 200.
e.g.
register = protobuf_pb2.register()
register.name = "foo"
register.id = 1
response = requests.request("POST", url, data=register.SerializeToString(), headers=headers)
when I look at this payload in SerializeToString() format it shows similar to
b'\n\t\n\x03foo\x10\x01'
As a sanity check, I can then create a new message object and .ParseFromString() on that back into the correct message format.
A second uri on the server allows you to send a GET request to retrieve that same data. So I used the following:
register = protobuf_pb2.register()
response = requests.request("GET", url, headers=headers).content
register.ParseFromString(response)
The response returns
b'foo:1\n'
And ParseFromString returns the exception
DecodeError('Error parsing message',)
Am I missing a step somewhere?
Should the .content payload be encoded somehow before being parsed?
Many thanks.
The object you are getting back from the server is binary data, UTF-8 encoded, so Python interprets it as a bytes object (in Python 3, bytes and str are different types).
m = b'foo:1\n'
type(m)
Out[1]: bytes
m.decode('utf-8')
Out[2]: 'foo:1\n'
ParseFromString however expects a string, so it throws a DecodeError; just use the decode() with the appropriate encoding.
In the response headers you should be able to find the encoding used: pretty much everyone uses UTF-8, but in "real" Production code, you should check that too: in the Content-Type (or you can even request a specific one with Accept-Charset but you may get a 406, if the server cannot fulfil it).
Related
body_1 = {"landowner": "asdf#gmail.com","title": report_name,"content": "asdf","project": projectID, }
url_1 = 'http://' + IP + ':8005/projects/notifications/'
response_1 = requests.post(url_1, data=body_1, headers=headers)
For this, I am getting an error 500, when above I have used a similar process to post and it has worked.
I even wrote this code in javascript, and that works but the python equivalent does not.
I tried to add console logs, and everything works up until the response, where it says code 500.
I tried to follow along with javascript as well and try and model it after that too, but that did not work either.
You are getting the error because you are not sending the data as JSON encoded data. When sending an array as data with a request use the json keyword argument. This means your request will look like;
response_1 = requests.post(url_1, json=body_1, headers=headers)
In the example above data= was changed to json=.
By using the json argument the data is encoded to JSON, and the Content-Type header is set to application/json.
Learn more about the json= keyword argument in the requests documentation
If this doesn't fix your issue please add the error output message for the 500 error code.
I have tested some requests inside the Postman app. First, I want to get the body information of an HTTP request inside Python (package requests used). The response appears positive with 200 OK.
response = session.request("POST", url, headers=headers, data=payload, verify ='custom-proxy-ca.crt')
Now I would like to get the body with
body = response.content
Print(body) delivers
b'\x83\x84\x01\x00\xc4\xff\xd4\xe9\xb4\xf6\xde,\x13\xa9\xc0(\xc7_\x8dL\x90\xf0\xb4K\xc4<\xe7\xb1M\x02)\xe0\x80z\xd0\xdf>\xcf\xd7\xd2\xec\x8d\x1e\xe4un\x0c\x83\xa1\x88g\xe7fah\x89\xbe\xca\xa8\x04_\xa2W\xbd\xfe]W\xd1\x06\x1f\xef~ZN\xa6\x0bq\xfa\x18\xc4\x1f\xb3\xf8\xc2\x9dF\xc5\xf0\xe6\x8d\xb6\xc1\xa0\xab\x7f\xfbyM\xe0\x88I\xb4\xd4\x82\xa1%\xd9R7Nt\xa4~<\x8c\x8e\xdb\xe7<xx-.\xab\xa7|16\xcb"\xba\x89\xbc\xe7\xcaF\xd1\xacV-u\xbf\xaa\x04\xf7\xa2\x88\xa1\x1bUI\xdfkI$`\x18:j\x7fU\x02\x0e\xcb\x97\x8em\xc6\x81\xe6\x85\xbe\xa5\xb9vbjQ$}M&n\xe0$A\xe0\xd9\xd2\xc6\x9aA\xf4\x12\x81/1\x0c\xf0(\x0cy\xf5\xaf\xca\x1bQ\x1082\xa1\xb4n4VRR\xbb7\xa5XO\x08\x0c\x13\xf2:\xc0-\x06\xa9\xda\xaeGX\x97B\x81!\x17\x87\xfa\xd1\x1b\xc0\xd0\x89|\xe8E\x0f\rp\xfd\x00\x96\xeaI\xbe\xda\xbb\xe3\x87\xc7\xdb\x9b\xfd\xab\xe8\xc7\xdd\x0cEL-x\xe0\x9bVhY\x0cT\x08\x95S\xa3\xfd\xdc\xe3\x81/1\x9d\x9e\'T\xf6\xe0pl\xd33#0,T}X%\x04\x0e\xd7r\xfd\x10\x0cs\xe90\x05\xe8\xe8\xf8\xea\xfc\xe5\xf8\xe1\xfd\xb9\xea\xe7\xe0\xc0\x9a!\xa1\\M\xa8\x9d\x9f\xe4\xa2\x07_\xae\xd7\x0c\xdd\xb8\xaa\xbf\xe9\xfc\x1a|\x89^\xf59\x81\xe3J\x91\xa4v(\xff7J1\x1ao\x9c\x89\xa1#0\xf4\xaa\xa0\xc7\xbc\xea\x9f\xae\xa6\xe8\xa9-T\xc9#\xd1\x81\x7f\xee\x9a\xbb\xfd\x87\xc3\xe3+|K\xe2\xfdPe\xa0\xaa\x9d\x18\xf0\xcc\xc0\xf10\x80\xca\xb0XuW\x9d\xcc\xc0\xa5\xc8;bP\xdd\x9d\x1aeC\xfd\xf84\xa6\x14yG\xeb\xb5\x01\x03'
Now I try to search a token in the body, but it seems to be encrypted.
If I want to get the result of the JSON parser with
json.loads(body)
it returns
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte`.
Okay, it seems that the encoding is done in a different way than expected. But how did the Postman app do the decoding of the body? For example, I can read it there parsed as JSON (see the figure below). What am I doing wrong in Python?
Request
Okay, the problem is solved, but I want to share with you how to deal with this kind of problem.
The initial problem is to call the HTTP POST request with the header parameter Accept-Encoding like
'Accept-Encoding': 'gzip, deflate, br'
This line of code means: Locally can receive data in compressed format.
The server compresses the large file and sends it back to the client during processing. After receiving the IE, the IE performs a local pressure on the file.
The reason for the error is: the program did not extract the file
Solution: delete this line of code and it works
I am trying to get a json object returned from my api. Using python's request framework to GET a json object from the api. Content type returns application/json when run so the content is json.
url = 'theUrl'
response = requests.get(url)
print(response.headers['content-type'])
data = json.load(response)
The script when ran returns:
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded.
There's also another issue when carrying out the request the api throws an error. The api's error log produces an error of
message.internal.WriterInterceptorExecutor:MessageBodyWriter not found for media type={text/html}
I've tried various different approaches to this but can't seem to receive the json object. Anyone encounter similar issues?
The last two lines make no sense. You call response.json() and then ignore the return value, and then try to call json.loads() on the response itself, rather than the actual response content.
Instead of those two lines, just do data = response.json() which already returns the deserialized data.
The return value from requests.get is not a string but an object. The response body is in .text property, thus:
data = json.loads(response.text)
And you are done
I've solved my issue, but I'd like to know what was going wrong so I can address it in the future. I'm having issues decoding incoming JSON for use in my Flask application.
The code that sends it in Angular:
$http.post("/login", JSON.stringify($scope.loginForm))
.success(function(data, status, headers, config) {
console.log(data);
})
.error(function(data, status, headers, config) {
console.log("Submitting form failed!");
});
Important to note that the request type is set to application/json earlier up, with
$http.defaults.headers.post["Content-Type"] = "application/json";
The code that receives it within Flask:
data = request.get_json()
email_address = data.get("email_address")
password = data.get("password")
Attempting to load it this way returns an error 400, but any other way leads to some very strange issues. For example:
return json.dumps(request.get_json())
Will log {"password": "password", "email_address": "email#email.com"} in the console, but attempting to do this:
data = request.get_json()
email_address = data.get("email_address")
password = data.get("password")
With no difference whatsoever between this and the first block of code except that I'm not forcing it, I receive the exception "ValueError: need more than 1 value to unpack". Which implies that there aren't two values to unpack.
HOWEVER, they both work individually. If I do the above request and omit either of the data.get() lines above, the other will work.
What about my setup causes my JSON object to disintegrate the first time it's accessed?
I got around this by using request.json instead of request.get_json() but as request.json is being deprecated it's fairly important I know how to solve this in the future. Any pointers would be appreciated!
You can omit JSON.stringify and pass object directly to $http.post() method because angular will serialize it to JSON automatically it formData is object. So I assume that JSON.stringify will force angular to send is as x-www-form-urlencoded instead of application/json media type.
See default transformations section: angular $http service documentation
I'm trying to re-create the example available on this page
Of course I'm changing the client_id, secret, credit card etc... with my valid data (I haven't copy-pasted the example as is).
You can see my complete code here (I've hidden sensible data with *** ).
I can get the token without any problem, but when I post the payment request I get this back:
In [11]: r2.text
Out[11]: u'{"name":"MALFORMED_REQUEST","message":"The request JSON is not well formed.","information_link":"https://developer.paypal.com/webapps/developer/docs/api/#MALFORMED_REQUEST","debug_id":"*************"}'
I really can't understand why it says that my json is malformed :(
Anyone can help me? Thanks!
This is your code:
post_data = json.loads(s)
r2 = requests.post('https://api.sandbox.paypal.com/v1/payments/payment', headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer ' + token},
data = post_data)
You have a JSON string s that you convert to a Python object and post that to the server. As the docs say, if you pass a dict as your data, it gets form-encoded, not JSON-encoded. If you want data in any other format, you're supposed to encode it yourself, as in the example:
>>> r = requests.post(url, data=json.dumps(payload))
Since you already have the JSON-encoded string s, you can just send that.
You totally misunderstood what you should pass to requests.post() method. http://docs.python-requests.org/en/latest/user/quickstart.html#more-complicated-post-requests
You are trying to pass Python objects instead of JSON. #Janne's answer is an option, but it is more convenient to build data as Python object and then use json.dumps(obj) and pass result to requests.post().