In my post handler, the body of the message is a json.
In self.request.body, I'm getting a HTML encoded message:
%7B+%22name%22%3A+%22John+Dao%22%2C+%22Age%22%3A+42+%7D=
Taking a look at the network traffic, the payload is actually:
{ "name": "John Dao", "Age": 42 }
So, I'm pretty sure the encoding happens on the server. How do I decode this string, or somehow tell json.loads to accept encoded message, or better yet - tell WSGI (It is Google App Engine) not encoding the body to beging with?
>>> import urllib
>>> urllib.unquote_plus("%7B+%22name%22%3A+%22John+Dao%22%2C+%22Age%22%3A+42+%7D=")
'{ "name": "John Dao", "Age": 42 }='
It looks as though the GAE implementation of WebOb is trying to parse and rewrite the POST body as though its content type is "application/x-www-urlencoded" - even the url-decoded string has a "=" appended to it.
If this is the case, and you can change the client behavior, try setting it to something like "application/json" instead.
If you want to get the values corresponding to the keys name and Age, you can simple call self.request.get('name') and self.request.get('Age'). If the key is not found it will return an empty string by default.
Related
I am working with webhooks from Bold Commerce, which are validated using a hash of the timestamp and the body of the webhook, with a secret key as the signing key. The headers from the webhook looks like this :
X-Bold-Signature: 06cc9aab9fd856bdc326f21d54a23e62441adb5966182e784db47ab4f2568231
timestamp: 1556410547
Content-Type: application/json
charset: utf-8
According to their documentation, the hash is built like so (in PHP):
$now = time(); // current unix timestamp
$json = json_encode($payload, JSON_FORCE_OBJECT);
$signature = hash_hmac('sha256', $now.'.'.$json, $signingKey);
I am trying to recreate the same hash using python, and I am always getting the wrong value for the hash. I've tried several combinations, with and without base64 encoding. In python3, a bytes object is expected for the hmac, so I need to encode everything before I can compare it. At this point my code looks like so :
json_loaded = json.loads(request.body)
json_dumped = json.dumps(json_loaded)
# if I dont load and then dump the json, the message has escaped \n characters in it
message = timestamp + '.' + json_dumped
# => 1556410547.{"event_type" : "order.created", "date": "2020-06-08".....}
hash = hmac.new(secret.encode(), message.encode(), hashlib.sha256)
hex_digest = hash.hexdigest()
# => 3e4520c869552a282ed29b6523eecbd591fc19d1a3f9e4933e03ae8ce3f77bd4
# hmac_to_verify = 06cc9aab9fd856bdc326f21d54a23e62441adb5966182e784db47ab4f2568231
return hmac.compare_digest(hex_digest, hmac_to_verify)
Im not sure what I am doing wrong. For the other webhooks I am validating, I used base64 encoding, but it seems like here, that hasnt been used on the PHP side. I am not so familiar with PHP so maybe there is something I've missed in how they built the orginal hash. There could be complications coming from the fact that I have to convert back and forth between byte arrays and strings, maybe I am using the wrong encoding for that ? Please someone help, I feel like I've tried every combination and am at a loss.
EDIT : Tried this solution by leaving the body without encoding it in json and it still fails :
print(type(timestamp)
# => <class 'str'>
print(type(body))
# => <class 'bytes'>
# cant concatenate bytes to string...
message = timestamp.encode('utf-8') + b'.' + body
# => b'1556410547.{\n "event_type": "order.created",\n "event_time": "2020-06-08 11:16:04",\n ...
hash = hmac.new(secret.encode(), message, hashlib.sha256)
hex_digest = hash.hexdigest()
# ...etc
EDIT EDIT :
Actually it is working in production ! Thanks to the solution described above (concatenating everything as bytes). My Postman request with the faked webhook was still failing, but that's probably because of how I copied and pasted the webhook data from my heroku logs :) .
I'm working on a Microservices Project, I'm using Python as a language and RabbitMQ as Message broker. I'm working on an App Store application and I'm facing this problem that the frontend should send me some data in a JSON which include for example (name of the app -> str, title of the app -> str, image or images of the app -> image ....) and that will be sent as a Request over RabbitMQ, the backend should consume that Request and store those data in a Mongodb database and then return a Response for example {"Success": true}.
I've tried so many things, I figured out how to store an Image in the Database. What I didn't figure out is how to send that Image data in a JSON. I tried base64 encoding, utf8, latin1 but None of it works because type bytes is not JSON serializable as the Error said. I searched a lot in the Internet but I didn't find exactly what I'm looking for although sometimes there is a similar questions but I tried almost everything and they didn't provide what I'm looking for.
data = {
"tool_owner": "John",
"developer": "John",
"icon": # icon of the app,
"images": # list of images,
"rating": {"user": "daniel", "date": str(now), "number": 5},
}
d = json.dumps(data) # this is my goal. to send the data as a JSON
temp = json.loads(d) # and receive it in the backend and parse it so I
can retrieve the data and store it in mongodb
in the first time, I face this error:
TypeError: Object of type bytes is not JSON serializable.
I tried to look how can I convert bytes to strings so that I can send it in a JSON but what I found didn't work for me since I tried base64 encoding and latin1 encoding...
Your dictionary must contain bytes objects, which is not supported by the json standard.
You can use jsonextra for serialization/deserialization to overcome this issue.
$ pip install jsonextra
import jsonextra as json
d = {...} # contains bytes objects
x = json.dumps(d)
y = json.loads(x)
assert d == y
See https://github.com/den4uk/jsonextra for more information.
I tested using Python to translate a curl to get some data.
import requests
import json
username="abc"
password="123"
headers = {
'Content-Type': 'application/json',
}
params = (
('version', '2017-05-01'),
)
data = '{"text":["This is message one."], "id":"en-es"}'
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, data=data, auth=(username, password))
print(response.text)
The above works fine. It returns json data.
It seems ["This is message one."] is a list. I want to use a variable that loads a file to replace this list.
I tried:
with open(f,"r",encoding='utf-8') as fp:
file_in_list=fp.read().splitlines()
toStr=str(file_in_list)
data = '{"text":'+toStr+', "id":"en-es"}'
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, data=data, auth=(username, password))
print(response.text)
But it returned error below.
{
"code" : 400,
"error" : "Mapping error, invalid JSON"
}
Can you help? How can I have valid response.text?
Thanks.
update:
The content of f contains only five lines below:
This is message one.
this is 2.
this is three.
this is four.
this is five.
The reason your existing code fails is that str applied to a list of strings will only rarely give you valid JSON. They're not intended to do the same thing. JSON only allows double-quoted strings; Python allows both single- and double-quoted strings. And, unless your strings all happen to include ' characters, Python will render them with single quotes:
>>> print(["abc'def"]) # gives you valid JSON, but only by accident
["abc'def"]
>>> print(["abc"]) # does not give you valid JSON
['abc']
If you want to get the valid JSON encoding of a list of strings, don't try to trick str into giving you valid JSON by accident, just use the json module:
toStr = json.dumps(file_in_list)
But, even more simply, you shouldn't be trying to figure out how to construct JSON strings in the first place. Just create a dict and json.dumps the whole thing:
data = {"text": file_in_list, "id": "en-es"}
data_str = json.dumps(data)
Being able to do this is pretty much the whole point of JSON: it's a simple way to automatically serialize all of the types that are common to all the major scripting languages.
Or, even better, let requests do it for you by passing a json argument instead of a data argument:
data = {"text": file_in_list, "id": "en-es"}
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, json=data, auth=(username, password))
This also automatically takes care of setting the Content-Type header to application/json for you. You weren't doing that—and, while many servers will accept your input without it, it's illegal, and some servers will not allow it.
For more details, see the section More complicated POST requests in the requests docs. But there really aren't many more details.
tldr;
toStr = json.dumps(file_in_list)
Explanation
Assuming your file contains something like
String_A
String_B
You need to ensure that toStr is:
Enclosed by [ and ]
Every String in the list is enclosed by quotation marks.
So your raw json (as a String) is equal to '{"text":["String_A", "String_B"], "id":"en-es"}'
i am trying to create bill for payment and send to my customer via telegram bot:
i am using blockchain API V2-https://blockchain.info/api/api receive .my code is:
xpub='***'
keyk='02e57f1***'
url='https://api.blockchain.info/v2/receive?xpub='+str(xpub)+'&callback=https%3A%2F%2Fdoors03.ru&key='+keyk
x=requests.get(url)
r=x.json()
r=r['address']
r -is an adress wich was made.
i am sending it to my costumer(by the way is there any way to send adress with exact sum for pay ) . After i want to check is payment was recieved:
data={ "Content-Type": "text/plain","key":keyk,"addr":r,"callback":"https%3A%2F%2Fdoors03.ru","onNotification":"KEEP", "op":"RECEIVE"}
r = requests.post(url, data=data)
and this is the response - u'{\n "message" : "Internal handlers error"\n}'
what i am doing wrong ? how to check payments ? how to send address with exact sum of btc or ethereum ?
Sorry, i don't have enough reputation to post a comment, so this is
the only option i have. #egorkh have you solved this problem? Maybe
you have received explanation from blockchain.info support? I have
sent them a question about that, but they are answering for too long.
UPDATE: Finally, i have found solution.
In my case, reason of "Internal handlers error" message is in a wrong interpretation of their API.
As they haven't implemented balance_update request in their java-api, i did it on my own and i did it in wrong way.
I have put this parameters:
{"key":keyk,"addr":r,"callback":"https%3A%2F%2Fdoors03.ru","onNotification":"KEEP", "op":"RECEIVE"}
as post parameters, like in other methods they have provided in api. In those methods parameters are URLEncoded like you did with callback link. But...
In this HTML request they must be sent as plain text in json format without any special encoding, like that:
Map<String, String> params = new HashMap<String, String>();
params.put("addr", address);
params.put("callback", callbackUrl);
params.put("key", apiCode);
params.put("onNotification", keepOnNotification? "KEEP" : "DELETE");
params.put("confs", Integer.toString(confirmationCount));
params.put("op", StringUtils.isBlank(operationType) ? "ALL" : operationType);
//parse parameters map to json string(that's optional: you can write it directly as string)
String body = new Gson().toJson(params);
if (requestMethod.equals("POST")) {
byte[] postBytes = body.getBytes("UTF-8");
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type", "text/plain");
conn.setRequestProperty("Content-Length", String.valueOf(postBytes.length));
conn.getOutputStream().write(postBytes);
conn.getOutputStream().close();
}
The main reason of your error may be that you put "Content-Type": "text/plain" in data object (, and maybe encoded callback url) .
I am trying to save a python dictionary in a cookie and then be able to convert the cookie value back to a dict in jquery or javascript. I'm using python 3.3.1. I'm getting confused on formats ( query strings, json, etc).
I'm using this jquery cookie library
https://github.com/carhartl/jquery-cookie
I have tried the following:
server side:
import urlib.parse
d= {'key1':'value1','key2':'value2'}
cookie_data = urllib.parse.urlencode(d)
client side:
cookie_data = $.cookie('Cookie_Name');
var dict = $.parseJson(cookie_data);
What is the correct way of doing it so I end up with a valid dictionary on the client side?
I also have tried using:
server side:
d = {'key1':'value1','key2':'value2'}
cookie_data = json.dumps(d)
Which gets me the desired json data in a string.
cookie_data now is '{"key1":"value1","key2":"value2"}'
However when I send it as a cookie the string comes out like this
"\173\"key2\": \"value2\"\054 \"key1\": \"value1\"\175"
That's why I thought I needed to encode it somehow.
I have also tried:
var cookie_str = $.cookie('My_cookie');
var decoded_cookie = decodeURIComponent(cookie_str);
both cookie_str and decoded_cookie are the same
\173"key2": "value2"\054 "key1": "value1"\175
I figured it out by doing this server side:
import urlib.parse, json
cookie_data = urllib.parse.quote(json.dumps(mydict))
client side using jquery
var dict = $.parseJSON($.cookie('MyCookie'));
Using console logging in JS can be very useful to debug what info is being sent back and forth.
console.log(cookie_data)
will print what is actually stored in $.cookie('Cookie_Name');
The problem here is that you're storing url-encoded values in the server: key1=value1&key2=value2, but you want the json: {"key1": "value1", "key2": "value2"}. You need to encode the data into JSON format on the server, as that's what the client is expecting.
repr will get you close, but it uses single-quotes and the JSON standard specifies double-quotes for strings. Your client may accept this, but to conform to the standard, as well as escaping other data, you should use a python library for encoding JSON, such as simplejson:
>>> simplejson.JSONEncoder().encode({'k':'v', 'k2': 'v2'})
'{"k2": "v2", "k": "v"}'
>>> repr({'k':'v', 'k2': 'v2'})
"{'k2': 'v2', 'k': 'v'}"