The request I am trying to intercept and modify is a get request with only one parameter and I try to modify it:
from mitmproxy import http
def request(flow: http.HTTPFlow) -> None:
if flow.request.pretty_url.startswith(BASE_URL):
flow.request.url = BASE_URL.replace('abc', 'def')
The above shows what I am trying to do in a nutshell. But unfortunately, according to docs,
this event fires after the entire body has been streamed.
In the end, I am not able to modify the request. Am I missing something here? Because if modify requests is not possible, then what is the point of mitmproxy?
Related
I am trying to intercept and modify a graphql response's body. Here is my addon code:
from mitmproxy import ctx
from mitmproxy import http
import json
def response(flow: http.HTTPFlow) -> None:
if flow.request.pretty_url == "https://my.graphql/endpoint":
request_data = json.loads(flow.request.get_text())
if request_data["operationName"] == "MyOperationName":
data = json.loads(flow.response.get_text())
data["data"]["product"]["name"] = "New Name"
flow.response.text = json.dumps(data)
I can see the modified response in mitmproxy console. But the iOS simulator I am using is still getting the original response. Does anyone know how can I pass the modified response to the device?
From the documentation
def response(self, flow: mitmproxy.http.HTTPFlow):
"""
The full HTTP response has been read.
Note: If response streaming is active, this event fires after the entire body has been streamed.
HTTP trailers, if present, have not been transmitted to the client yet and can still be modified.
"""
ctx.log(f"response: {flow=}")
It appears that you might be streaming the response body which would mean that modifications would be ignored.
Consider using def request event hook instead
Good day,
I am currently trying to figure out a way to make non blocking requests inside a simple script of mitmproxy, but the documentation doesn't seem to be clear for me for the first look.
I think it's probably the easiest if I show my current code and describe my issue below:
from copy import copy
from mitmproxy import http
def request(flow: http.HTTPFlow):
headers = copy(flow.request.headers)
headers.update({"Authorization": "<removed>", "Requested-URI": flow.request.pretty_url})
req = http.HTTPRequest(
first_line_format="origin_form",
scheme=flow.request.scheme,
port=443,
path="/",
http_version=flow.request.http_version,
content=flow.request.content,
host="my.api.xyz",
headers=headers,
method=flow.request.method
)
print(req.get_text())
flow.response = http.HTTPResponse.make(
200, req.content,
)
Basically I would like to intercept any HTTP(S) request done and make a non blocking request to an API endpoint at https://my.api.xyz/ which should take all original headers and return a png screenshot of the originally requested URL.
However the code above produces an empty content and the print returns nothing either.
My issue seems to be related to: mtmproxy http get request in script and Resubmitting a request from a response in mitmproxy but I still couldn't figure out a proper way of sending requests inside mitmproxy.
The following piece of code probably does what you are looking for:
from copy import copy
from mitmproxy import http
from mitmproxy import ctx
from mitmproxy.addons import clientplayback
def request(flow: http.HTTPFlow):
ctx.log.info("Inside request")
if hasattr(flow.request, 'is_custom'):
return
headers = copy(flow.request.headers)
headers.update({"Authorization": "<removed>", "Requested-URI": flow.request.pretty_url})
req = http.HTTPRequest(
first_line_format="origin_form",
scheme='http',
port=8000,
path="/",
http_version=flow.request.http_version,
content=flow.request.content,
host="localhost",
headers=headers,
method=flow.request.method
)
req.is_custom = True
playback = ctx.master.addons.get('clientplayback')
f = flow.copy()
f.request = req
playback.start_replay([f])
It uses the clientplayback addon in order to send out the request. When this new request is sent, that will generate another request event which will then be an infinite loop. That is the reason for the is_custom attribute I added to the request there. If the request that generated this event is the one that we have created, then we don't want to create a new request from it.
I tried mitmproxy in the last couple of days as a test tool and works excellent. However, while I'm able to write add-ons that intercept requests (even changing their URL, like my example below), I couldn't avoid that the request is actually dispatched in the network.
One way or another, always the request is performed using the network.
So, how can I modify my add-on in a way that, giving a request, it returns a fixed response, avoiding any networking request?
class Interceptor:
def request(self, flow: http.HTTPFlow):
if http.method() == "GET":
flow.request.url = "http://google.com"
def response(self, flow: http.HTTPFlow):
return http.HTTPResponse.make(status_code=200,b"Rambo 5")
The request hook will be executed when mitmproxy has received the request, the response hook will be executed once we have fetched the response from the server. Long story short, everything in the response hook is too late.
Instead, you need to assign flow.response in the request hook.
I have seen Tornado documentations and examples where self.write method is widely used to render some value on HTML, where the POST request was run in a handler. But I could not find much clarity on how to return the response back to client.
For example, I am calling a POST request on a Tornado server from my client. The code that accepts post request is:
class strest(tornado.web.RequestHandler):
def post(self):
value = self.get_argument('key')
cbtp = cbt.main(value)
With this, I can find the value of cbtp and with self.write(cbtp), I can get it printed in HTML. But instead, I want to return this value to the client in JSON format, like {'cbtp':cbtp}
I want to know how to modify my code so that this response is sent to the client, or give me some documentation where this this is fluently explained.
Doing something like
res = {cbtp: cbtp}
return cbtp
throws a BadYieldError: yielded unknown object
You just need to set the output type as JSON and json.dumps your output.
Normally I have the set_default_headers in a parent class called RESTRequestHandler. If you want just one request that is returning JSON you can set the headers in the post call.
class strest(tornado.web.RequestHandler):
def set_default_headers(self):
self.set_header("Content-Type", 'application/json')
def post(self):
value = self.get_argument('key')
cbtp = cbt.main(value)
r = json.dumps({'cbtp': cbtp})
self.write(r)
If the given chunk is a dictionary, we write it as JSON and set the Content-Type of the response to be application/json. (if you want to send JSON as a different Content-Type, call set_header after calling write()).
Using it should give you exactly what you want:
self.write(json.dumps({'cbtp': cbtp}))
We have some custom module where we have redefined open, seek, read, tell functions to read only a part of file according to the arguments.
But, this logic overrides the default tell and python requests is trying to calculate the content-length which involves using tell(), which then redirects to our custom tell function and the logic is somewhere buggy and returns a wrong value. And I tried some changes, it throws error.
Found the following from models.py of requests:
def prepare_content_length(self, body):
if hasattr(body, 'seek') and hasattr(body, 'tell'):
body.seek(0, 2)
self.headers['Content-Length'] = builtin_str(body.tell())
body.seek(0, 0)
elif body is not None:
l = super_len(body)
if l:
self.headers['Content-Length'] = builtin_str(l)
elif (self.method not in ('GET', 'HEAD')) and (self.headers.get('Content-Length') is None):
self.headers['Content-Length'] = '0'
For now, I am not able to figure out where's the bug and stressed out to investigate more and fix it. And everything else work except content-length calculation by python requests.
So, I have created my own definition for finding content-length. And I have included the value in requests header. But, the request is still preparing the content-length and throwing error.
How can I restrict not preparing content-length and use the specified content-length?
Requests lets you modify a request before sending. See Prepared Requests.
For example:
from requests import Request, Session
s = Session()
req = Request('POST', url, data=data, headers=headers)
prepped = req.prepare()
# do something with prepped.headers
prepped.headers['Content-Length'] = your_custom_content_length_calculation()
resp = s.send(prepped, ...)
If your session has its own configuration (like cookie persistence or connection-pooling), then you should use s.prepare_request(req) instead of req.prepare().