Develop Reference

How to make a request inside a simple mitmproxy script?

Good day,
I am currently trying to figure out a way to make non blocking requests inside a simple script of mitmproxy, but the documentation doesn't seem to be clear for me for the first look.
I think it's probably the easiest if I show my current code and describe my issue below:
from copy import copy
from mitmproxy import http
def request(flow: http.HTTPFlow):
headers = copy(flow.request.headers)
headers.update({"Authorization": "<removed>", "Requested-URI": flow.request.pretty_url})
req = http.HTTPRequest(
first_line_format="origin_form",
scheme=flow.request.scheme,
port=443,
path="/",
http_version=flow.request.http_version,
content=flow.request.content,
host="my.api.xyz",
headers=headers,
method=flow.request.method
)
print(req.get_text())
flow.response = http.HTTPResponse.make(
200, req.content,
)
Basically I would like to intercept any HTTP(S) request done and make a non blocking request to an API endpoint at https://my.api.xyz/ which should take all original headers and return a png screenshot of the originally requested URL.
However the code above produces an empty content and the print returns nothing either.
My issue seems to be related to: mtmproxy http get request in script and Resubmitting a request from a response in mitmproxy but I still couldn't figure out a proper way of sending requests inside mitmproxy.

The following piece of code probably does what you are looking for:
from copy import copy
from mitmproxy import http
from mitmproxy import ctx
from mitmproxy.addons import clientplayback
def request(flow: http.HTTPFlow):
ctx.log.info("Inside request")
if hasattr(flow.request, 'is_custom'):
return
headers = copy(flow.request.headers)
headers.update({"Authorization": "<removed>", "Requested-URI": flow.request.pretty_url})
req = http.HTTPRequest(
first_line_format="origin_form",
scheme='http',
port=8000,
path="/",
http_version=flow.request.http_version,
content=flow.request.content,
host="localhost",
headers=headers,
method=flow.request.method
)
req.is_custom = True
playback = ctx.master.addons.get('clientplayback')
f = flow.copy()
f.request = req
playback.start_replay([f])
It uses the clientplayback addon in order to send out the request. When this new request is sent, that will generate another request event which will then be an infinite loop. That is the reason for the is_custom attribute I added to the request there. If the request that generated this event is the one that we have created, then we don't want to create a new request from it.

how to create raw http responses in flask?

I am working on a software that has a TCP Server that replies to requests done in a proprietary protocol.
Obviously the implementation relies on a socket listening on a fixed port and on analyzing and managing raw request and response.
I should add to this service the possibility to manage http requests.
I started using flask with the idea to let it manage templates rendering and responses creation, but I am a bit struck on the second part:
Righ now I managed to make this work with something like this:
with open(template_file) as f:
template = f.read()
app = flask.Flask('my app') # create a context to render Response
with app.app_context():
context = {'title': 'mytitle',
'other_info':'.....',}
rendered = flask.render_template_string(template, **context)
response = flask.make_response(rendered)
answer = f'''HTTP/1.0 200 OK\nContent-Type: text/html\n\n {rendered} \n\n'''
sock.sendall(answer.encode())
sock.close()
In this case make_response creates a Response instance where you can get the rendered html code but going from Response to the raw http is my problem.
To solve this i have added manually a header, but I think that there is a better way in flask to do this but can't figure out.
To make the question more general: how can coexist flask web application with others? Where is the point on which I have to take control of the process?

Get url without blocking the function

I'm trying to get a resource from a url inside a route on a web server without blocking it since getting it sometimes takes 11 seconds+.
I switched from flask to aiohttp for this.
async def process(request):
data = await request.json()
req = urllib.request.Request(
request["resource_url"],
data=None,
headers=hdrs
)
# Do processing on the resource
But I'm not sure how to make the call and would it allow other calls to be made to this route while the resource is getting fetched?

Receive async requests python

I have several Flask servers which handle POST requests and returns some values. I need to run an infinite process which sends requests to all servers, waiting until first response, update internal state based on that response and send new request to that server again. Here is a pseudocode for this:
obj = SomeObject()
requests = [obj.make_request() for _ in range(10)]
responses = grequests.imap(requests, size=10)
for response in responses:
obj.update_state(response)
requests.append(obj.make_request())
What is the proper implementation of such logic in python?

Non blocking download of file in flask?

I am making a drop box like service using openstack. I am making a web interface using flask. User gets the object data in content of get request. I am sending data to the user iteratively. But my Flask app stops until the whole object is dowloaded. How could I make it non blocking?
#Returns the json content
r = swift_account.getObject(container_name, object_name)
filename = r.headers['X-Object-Meta-Orig-Filename']
#Make a generator so that all the content are not stored at once in memory
def generate():
for chunk in r.iter_content():
yield chunk
response = make_response(Response(stream_with_context(generate())))
response.headers['Content-Disposition'] = 'attachment; filename=' + filename
return response

Whether flask runs as blocking or non-blocking depends how you run it. When you say it blocks, how are you running it?
Unless it's being run by something with async capabilities like gunicorn with async or at least a threading model for multiple requests like apache with mod_wsgi then it won't be able to respond to more than one request at a time.