I tried mitmproxy in the last couple of days as a test tool and works excellent. However, while I'm able to write add-ons that intercept requests (even changing their URL, like my example below), I couldn't avoid that the request is actually dispatched in the network.
One way or another, always the request is performed using the network.
So, how can I modify my add-on in a way that, giving a request, it returns a fixed response, avoiding any networking request?
class Interceptor:
def request(self, flow: http.HTTPFlow):
if http.method() == "GET":
flow.request.url = "http://google.com"
def response(self, flow: http.HTTPFlow):
return http.HTTPResponse.make(status_code=200,b"Rambo 5")
The request hook will be executed when mitmproxy has received the request, the response hook will be executed once we have fetched the response from the server. Long story short, everything in the response hook is too late.
Instead, you need to assign flow.response in the request hook.
Related
The request I am trying to intercept and modify is a get request with only one parameter and I try to modify it:
from mitmproxy import http
def request(flow: http.HTTPFlow) -> None:
if flow.request.pretty_url.startswith(BASE_URL):
flow.request.url = BASE_URL.replace('abc', 'def')
The above shows what I am trying to do in a nutshell. But unfortunately, according to docs,
this event fires after the entire body has been streamed.
In the end, I am not able to modify the request. Am I missing something here? Because if modify requests is not possible, then what is the point of mitmproxy?
I have created a an application in python-django, where it makes some api call to some 3rd party services and then return response, post that I am doing some processing on response data and generating some final document. Below is something I am doing:
def get_processed_data(url, payload, tenant, req_id, timeout=None):
query_kwargs = HTTPRequestUtils.__get_query_kwargs(timeout)
query_kwargs['json'] = payload
response = HTTPRequestUtils.__get_response(url, query_kwargs, requests.post)
.....
data=process_response(response)
return more_processings(data)
Abobe is one of the function , being called during actual execution of code. And response varies with url.
Now problem is I have am writing Unit Test , and i have to emulate/mock http call, so that for different url, i may return different mocked response, that will be further processed.
I went through several libraries like responses etc, but what I conclude from them is , i can test just api call and its returned response. But in actual I need to just emulate/mock http call for different and return response back, like we do in patch during mock, so that the response can be further go for processing.
Any library or method by which we can achieve this.
If you have an idea of order in which the API call would take place, you can make use of side-effect func of mock library, so what it does is it will give different response for each time the mock function is called
for eg:
mock_api.side_effect = [(resp1),(resp2)]
so when api() will be called for the 1st time => response will be resp1 and for the second time ==> response will be resp2
I think this will solve your problem
I am using Python flask. I have a POST request with some payload coming on say:
abc.com/hello/hello1
I want to redirect this (302) to:
xyz.com/hello/hello1
only changing the domain name while keeping the remaining part as it is and also the payload. Is there a simple way to do this?
As per RFC, redirect requests (all 3xx) cannot contain request data or headers. You will miss the payload, supplied via POST in original request.
There are two possible workaround I could think of right away:
Give the client new URL, and implement further logic on client side;
Create a proxy handler on backend, which will do a request by itself and give the answer back as it's own.
EDIT: As per Andrejs Cainikovs's comment below, this would not work for a POST with payload.
In your endpoint, get the url that was used using request.url (see request API here for more options). Then you can rewrite it and make a redirect.
newUrl = "xyz.com/" + route
return redirect(newUrl, code=302)
I'm using YouTube data API v3.
Is it possible to make a big BatchHttpRequest (e.g., see here) and also to use ETags for local caching at the httplib2 level (e.g., see here)?
ETags work fine for single queries, I don't understand if they are useful also for batch requests.
TL;DR:
BatchHttpRequest cannot be used with caching
HERE IT IS:
First lets see the way to initialize BatchHttpRequest:
from apiclient.http import BatchHttpRequest
def list_animals(request_id, response, exception):
if exception is not None:
# Do something with the exception
pass
else:
# Do something with the response
pass
def list_farmers(request_id, response):
"""Do something with the farmers list response."""
pass
service = build('farm', 'v2')
batch = service.new_batch_http_request()
batch.add(service.animals().list(), callback=list_animals)
batch.add(service.farmers().list(), callback=list_farmers)
batch.execute(http=http)
Second lets see how ETags are used:
from google.appengine.api import memcache
http = httplib2.Http(cache=memcache)
Now lets analyze:
Observe the last line of BatchHttpRequest example: batch.execute(http=http), and now checking the source code for execute, it calls _refresh_and_apply_credentials, which applies the http object we pass it.
def _refresh_and_apply_credentials(self, request, http):
"""Refresh the credentials and apply to the request.
Args:
request: HttpRequest, the request.
http: httplib2.Http, the global http object for the batch.
"""
# For the credentials to refresh, but only once per refresh_token
# If there is no http per the request then refresh the http passed in
# via execute()
Which means, execute call which takes in http, can be passed the ETag http you would have created as:
http = httplib2.Http(cache=memcache)
# This would mean we would get the ETags cached http
batch.execute(http=http)
Update 1:
Could try with a custom object as well:
from googleapiclient.discovery_cache import DISCOVERY_DOC_MAX_AGE
from googleapiclient.discovery_cache.base import Cache
from googleapiclient.discovery_cache.file_cache import Cache as FileCache
custCache = FileCache(max_age=DISCOVERY_DOC_MAX_AGE)
http = httplib2.Http(cache=custCache)
# This would mean we would get the ETags cached http
batch.execute(http=http)
Because, this is just a hunch on the comment in http2 lib:
"""If 'cache' is a string then it is used as a directory name for
a disk cache. Otherwise it must be an object that supports the
same interface as FileCache.
Conclusion Update 2:
After again verifying the google-api-python source code, I see that, BatchHttpRequest is fixed with 'POST' request and has a content-type of multipart/mixed;.. - source code.
Giving a clue about the fact that, BatchHttpRequest is useful in order to POST data which is then processed down the later.
Now, keeping that in mind, observing what httplib2 request method uses: _updateCache only when following criteria are met:
Request is in ["GET", "HEAD"] or response.status == 303 or is a redirect request
ElSE -- response.status in [200, 203] and method in ["GET", "HEAD"]
OR -- if response.status == 304 and method == "GET"
This means, BatchHttpRequest cannot be used with caching.
I have a JavaScript bookmarklet that POSTs information to a (Flask powered) server while the user is on some other page (i.e. not one on my server). I don't want to interrupt the user's browsing by hijacking their session with my server response.
My initial thought was that I could suppress the HTTP response from Flask somehow; prevent it from sending anything to the client so they aren't mysteriously redirected. I was hoping I could do this by perhaps having a null return from a view.
I then thought that might be some HTTP response that lets the client know the information was successfully submitted, but will leave the client on their current page. Suppose a header value like "Here is the result of your request, but you should not alter your current display"?
To answer your amended question, yes there is such a response. From RFC 2616-section 10 (emphasis added):
10.2.5 204 No Content
The server has fulfilled the request but does not need to return an
entity-body, and might want to return updated metainformation. The
response MAY include new or updated metainformation in the form of
entity-headers, which if present SHOULD be associated with the
requested variant.
If the client is a user agent, it SHOULD NOT change its document view
from that which caused the request to be sent. This response is
primarily intended to allow input for actions to take place without
causing a change to the user agent's active document view, although
any new or updated metainformation SHOULD be applied to the document
currently in the user agent's active view.
The 204 response MUST NOT include a message-body, and thus is always
terminated by the first empty line after the header fields.
Thus from flask you can do something like this. Remember, the response must not include a message body, so any data you want to send back should be put into a cookie.
#app.route('/')
def index():
r = flask.Response()
r.set_cookie("My important cookie", value=some_cool_value)
return r, 204
No, it is not possible. Flask is built on Werkzeug, which implements the WSGI spec. The WSGI cycle requires sending a response to each request. Droping the response would require control over the TCP/IP connection at a far lower level even that HTTP. This is outside the domain of WSGI, therefore outside the domain of Flask.
You could return an error code, or an empty body, but you have to return something.
return '' # empty body