Develop Reference

How do I modify the JSON Body in a POST request with mitmproxy?

I am working with an app that sends data to a server with a POST request,
POST https://www.somedomain.com//sendImage HTTP/2.0
looking like this:
{
"user": {
"consent": true,
"currentNumberIs": 1,
"images": {
"data": "BASE64ENCODED IMAGE",
"docType": "avatar"
},
"totalNumberOfImages": 1
}
}
I want to replace the data part of this Json, but only if the docType is avatar. Trying to use a python script for that, that I found here and edited:
def response(flow: http.HTTPFlow) -> None:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
Launched mitmproxy with -s script.py, but according to the web console, the specific request does not trigger the script at all. Which kinda limits the scope to debug.
Would glady appreciate any help.

As #nneonneo mentioned in the comments, I would first recommend to make extensive use of mitmproxy.ctx.log() to make sure that your event hook is triggered properly. Second, if I understand things correctly, you intend to modify the request and not the response? If you want to modify request contents before they are sent to the server, you need to use the request hook and not the response hook:
def request(flow: http.HTTPFlow) -> None:
# this is executed after we have received the request
# from the client, but before it is sent to the server.
def response(flow: http.HTTPFlow) -> None:
# this is executed after we have sent the request
# to the server and received the response at the proxy.
Finally, you currently read from flow.request.text and then later assign to flow.response.text. I don't know your specific use case, but usually that should be flow.request.text as well.

You're altering the flow variable in a function, but not using the edited flow. If you return the new flow you can then use it and post it.
def response(flow: http.HTTPFlow) -> http.HTTPFlow:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
return flow

I don't know the cause and solution of keyerror

I'm a beginner.
What I used was flask and pymongo.
If you press the button, it's "Like". It should be +1, but there is a key error at the bottom.
My python route code:
#app.route('/api/like', methods=['POST'])
def like_movie():
title_receive = request.form['title_give']
movie = db.toytoy.find_one({'title': title_receive})
current_like = movie['like']
new_like = current_like + 1
db.toytoy.update_one({'title': title_receive}, {'$set': {'like': new_like}})
return jsonify({'msg': 'like!'})
This is how I POST from JS
function like_movie(title) {
$.ajax({
type: 'POST',
url: '/api/like',
data: {title_give: title},
success: function (response) {
console.log(response)
alert(response['msg']);
window.location.reload();
}
});
}
I get an exception as below:
werkzeug.exceptions.BadRequestKeyError: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
KeyError: 'title_give'
What I want is if it's 'like_btn'. If you press the button, it becomes +1.

The base problem in what you did is not respecting Content-type. From front JS, you are making a POST with JSON object. Which makes the request to have a content type of application/json.
In backend code, you use request.form which expects the request to be in the form encoded types (like application/x-www-form-urlencoded, multipart/form-data) etc.
So, you need to read the JSON content in backend, instead of reading from a form which is not available. Like below:
ui_req = request.get_json()
title_receive = ui_req['title_give']
And then parse other structures accordingly.

Alamofire 5 (Beta 6): Parameters of PUT Request do not arrive in Flask-Restful

UPDATE
For me the Problem got fixed as soon as I was putting "encoding: URLEncoding(destination: .queryString)" in my request. Maybe this helps somebody else. link
I struggled the whole day to find the problem in my Alamofire PUT Request or the Flask Restful API. Request like GET, DELETE and POST are working fine with Alamofire, except the PUT Request.
When I'm using PUT Requests in combination with Postman and Flask-Restful everything is also working fine. But as soon as I'm trying to achieve the same Result with Alamofire, I'm not getting any parameters in Flask. I tried to illustrate this in the code examples.
So in short my example illustrates the following:
DELETE Request(Same with GET and POST)
Postman: success
Alamofire: success
PUT Request
Postman: success
Alamofire: failure (parameter dictionary empty in Flask-Restful)
Here is my Python Code [API Server]:
from flask import Flask, request, jsonify
from flask_restful import Resource, Api, reqparse
app = Flask(__name__)
api = Api(app)
class Stackoverflow(Resource):
def delete(self):
print(request.args)
if request.args.get('test-key') is None:
return jsonify({"message": "failure"})
else:
return jsonify({"message": "success"})
def put(self):
print(request.args)
if request.args.get('test-key') is None:
return jsonify({"message": "failure"})
else:
return jsonify({"message": "success"})
api.add_resource(Stackoverflow, '/stackoverflow')
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
If I'm using Postman, I get this result (like expected):
Result in Postman
But now I'm trying to do the same with Alamofire in Swift. Same Server, nothing changed.
SWIFT demo Code [IOS APP]:
import UIKit
import Alamofire
import SwiftyJSON
class ViewController: UIViewController {
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view
simplePUTRequest()
simpleDELETERequest()
}
func simplePUTRequest(){
AF.request("http://localhost:5000/stackoverflow", method: .put, parameters: ["test-key":"testvalue"])
.validate(statusCode: 200..<300)
.responseJSON { response in
if let data = response.data {
print("Result PUT Request:")
print(String(data: data, encoding: .utf8)!)
//print(utf8Text)
}else{
}
}
}
func simpleDELETERequest(){
AF.request("http://localhost:5000/stackoverflow", method: .delete, parameters: ["test-key":"testvalue"])
.validate(statusCode: 200..<300)
.responseJSON { response in
if let data = response.data {
print("Result DELETE Request:")
print(String(data: data, encoding: .utf8)!)
//print(utf8Text)
}else{
}
}
}
Xcode Console:
Result PUT Request:
{
"message": "failure"
}
Result DELETE Request:
{
"message": "success"
}
python Console (both Alamofire Requests):
ImmutableMultiDict([])
127.0.0.1 - - [15/Jun/2019 21:17:31] "PUT /stackoverflow HTTP/1.1" 200 -
ImmutableMultiDict([('test-key', 'testvalue')])
127.0.0.1 - - [15/Jun/2019 21:17:31] "DELETE /stackoverflow?test-key=testvalue HTTP/1.1" 200 -
As you can see, I'm getting the success message only while using the DELETE method.
Till now I tried using different encodings like URLEncoding.httpbody and URLEncoding.default, but nothing really helped.
For me it seems like it's a Alamofire/Swift Problem, because in Postman the same request method is working fine.
I would really appreciate your help, because I'm stuck and don't know anything further to do. I hope I didn't misunderstood something essential.
Thank you in advance!

I am currently using the same version AlamoFire, and when I use the PUT method, I use it as follows:
let request = AF.request(url, method: .put, parameters: ["uid": uid],
encoding: JSONEncoding.default, headers: headers)
request.responseJSON(completionHandler: { response in
guard response.error == nil else {
//Handle error
}
if let json = response.value as? [String: Any]
// Handle result.
}
The only difference to your post is that I used the encoding option. You can try to put the option and see what happens.

It looks like your server is expecting your PUT parameters to be URL form encoded into the URL. You may be hitting the version of the request method that uses JSON encoding by default, so adding encoder: URLEncodedFormParameterEncoder.default at the end of your request call should fix that. A future release will make that the default, as it's safe across all request types.
If that's not the issue, I suggest you investigate more closely to see what the differences between the requests may be. Since you control the server you should have easy access to the traffic.

Kivy UrlRequest

My API works fine and I see a 200 status when I test it using Postman. However I'm trying to access it using a Kivy application but I'm seeing a 400 response from the server after some waiting or quitting the app. By the way when testing with Postman I specify header as Content-Type: application/json and in body I see my parameters
{
"search_text": "Hello",
"num_results": 1
}
being sent as raw data.
My code
def search(self, search_text):
header = {'Content-Type':'application/json'}
req = UrlRequest('http://127.0.0.1:5000/search',req_body={"search_text": search_text,"num_results": 1},on_success=Test.got_json,req_headers=header)
print("Search method called")
#staticmethod
def got_json(req,result):
print(result)
Kivy docs say that you don't have to specify a method as this would send a POST request so I've not specified that here
Edit: The code for the server is kind of irrelevant for my issue here so I've removed it

UrlRequest should be passed a str object as request body. You can serialize the request dictionary as a string object by dumping it. Pass this dumped dictionary as the request body to UrlRequest.
import json
req_body=json.dumps({'search_text': search_text, 'num_results': 1})
req = UrlRequest(
'http://127.0.0.1:5000/search',
req_body=req_body,
on_success=Test.got_json,
req_headers=header)

req_body is a string parameter, might be a bit confusing as req_headers is a dict. You can use:
req_body=json.dumps({"search_text": search_text,"num_results": 1})

Why can't I decode a JSON message in the following situation?

I am trying to send a JSON message from a computer to another one via a post request.
The script which sends the message is the following:
message = {'station':'turn on'}
res = rest.send( 'POST', server_addr + "/newstation", json.dumps(message), {'Content-Type': 'application/json'} )
The rest.send(...) method should be correct as I used it before and it worked fine.
The PC which sends the post request runs Linux, while the receiving one runs Win 8, if that means anything.
On the receiving machine I have the following:
#app.route('/newstation', methods = ['POST'])
def new_station ():
j_data = request.get_json()
d = decode_data(j_data)
where decode_data(j_data) is the following
def decode_data(j_data):
d = json.loads(j_data)
return d
My problem is: whenever I try to send the post request from the first machine the response is "Internal server error" and on the machine with the server the error returned is "TypeError: expected string or buffer".
Now I am thinking that it may be a matter of encoding of the string.
The post request is received and I can print the json content without problems, the issue arises when I try to decode.

I fixed the issue, it was a mistake on my part (of course). I misunderstood the documentation.
#app.route('/newstation', methods = ['POST'])
def new_station ():
j_data = request.get_json()
#d = decode_data(j_data)
request.get_json() already returns me a dictionary, so the decode_data function isn't actually needed. I already have the result without the need for json.loads().