I'm trying to get a resource from a url inside a route on a web server without blocking it since getting it sometimes takes 11 seconds+.
I switched from flask to aiohttp for this.
async def process(request):
data = await request.json()
req = urllib.request.Request(
request["resource_url"],
data=None,
headers=hdrs
)
# Do processing on the resource
But I'm not sure how to make the call and would it allow other calls to be made to this route while the resource is getting fetched?
Related
I'm migrating a program that's using requests to using aiohttp but I've hit a part of it were the response I get from using aiohttp is different to that from requests.
Function goes like this and tries to login a user to a website.
async def LoginLN(usuario, contraseña, email, url, session):
s = session # aiohttp session
# some stuff and some other requests using the same aiohttp session to get the appropriate cookies for login and up to this part, all responses are OK
r5 = await s.post('https://some-website.com.ar/register-user', headers=headerss5, cookies=cookie_dict, timeout = 140)
texto5 = await r5.text()
# some more stuff
And I get
texto5 = '{"registration_status":"invalid_credentials","user":[]}'
And user doesn't log in.
If I just replace the r5 definition by keeping every parameter the same to
async def LoginLN(usuario, contraseña, email, url, session):
# some stuff and some other requests using the same aiohttp session to get the appropriate cookies for login and up to this part, all responses are OK
ss = requests.session()
r5 = ss.post('https://some-website.com.ar/register-user', headers=headerss5, cookies = cookie_dict, timeout = 140)
texto5 = r5.text
# some more stuff
The response is the correct one and I'm able to login the user correctly. headerss5 and cookie_dict are usual header and cookie dictionaries.
I've wiresharked the two requests and see some differences but don't know how to further analyze this and even if I did find something I wouldn't know what to do about it to make aiohttp work properly but any suggestion is welcomed.
requests request
aiohttp request
I'm using windows 10 and:
requests == 2.27.1
aiohttp == 3.8.3
python == 3.10.0
I could finally see what was going on. Inspecting both requests (requests vs aiohttp) with mitmproxy I could see that the cookies sent by aiohttp had double quotes added where requests wouldn't add them generating the different response. Looking into this problem to find an already existing solution I followed the workaround proposed here of putting the cookies explicitly as header.
Good day,
I am currently trying to figure out a way to make non blocking requests inside a simple script of mitmproxy, but the documentation doesn't seem to be clear for me for the first look.
I think it's probably the easiest if I show my current code and describe my issue below:
from copy import copy
from mitmproxy import http
def request(flow: http.HTTPFlow):
headers = copy(flow.request.headers)
headers.update({"Authorization": "<removed>", "Requested-URI": flow.request.pretty_url})
req = http.HTTPRequest(
first_line_format="origin_form",
scheme=flow.request.scheme,
port=443,
path="/",
http_version=flow.request.http_version,
content=flow.request.content,
host="my.api.xyz",
headers=headers,
method=flow.request.method
)
print(req.get_text())
flow.response = http.HTTPResponse.make(
200, req.content,
)
Basically I would like to intercept any HTTP(S) request done and make a non blocking request to an API endpoint at https://my.api.xyz/ which should take all original headers and return a png screenshot of the originally requested URL.
However the code above produces an empty content and the print returns nothing either.
My issue seems to be related to: mtmproxy http get request in script and Resubmitting a request from a response in mitmproxy but I still couldn't figure out a proper way of sending requests inside mitmproxy.
The following piece of code probably does what you are looking for:
from copy import copy
from mitmproxy import http
from mitmproxy import ctx
from mitmproxy.addons import clientplayback
def request(flow: http.HTTPFlow):
ctx.log.info("Inside request")
if hasattr(flow.request, 'is_custom'):
return
headers = copy(flow.request.headers)
headers.update({"Authorization": "<removed>", "Requested-URI": flow.request.pretty_url})
req = http.HTTPRequest(
first_line_format="origin_form",
scheme='http',
port=8000,
path="/",
http_version=flow.request.http_version,
content=flow.request.content,
host="localhost",
headers=headers,
method=flow.request.method
)
req.is_custom = True
playback = ctx.master.addons.get('clientplayback')
f = flow.copy()
f.request = req
playback.start_replay([f])
It uses the clientplayback addon in order to send out the request. When this new request is sent, that will generate another request event which will then be an infinite loop. That is the reason for the is_custom attribute I added to the request there. If the request that generated this event is the one that we have created, then we don't want to create a new request from it.
I have several Flask servers which handle POST requests and returns some values. I need to run an infinite process which sends requests to all servers, waiting until first response, update internal state based on that response and send new request to that server again. Here is a pseudocode for this:
obj = SomeObject()
requests = [obj.make_request() for _ in range(10)]
responses = grequests.imap(requests, size=10)
for response in responses:
obj.update_state(response)
requests.append(obj.make_request())
What is the proper implementation of such logic in python?
This question already has answers here:
Flask hangs when sending a post request to itself
(2 answers)
Closed 5 years ago.
I was trying develop web application using flask, below is my code,
from sample import APIAccessor
#API
#app.route('/test/triggerSecCall',methods=['GET'])
def triggerMain():
resp = APIAccessor().trigger2()
return Response(json.dumps(resp), mimetype='application/json')
#app.route('/test/seccall',methods=['GET'])
def triggerSub():
resp = {'data':'called second method'}
return Response(json.dumps(resp), mimetype='application/json')
And my trigger method contains the following code,
def trigger2(self):
url = 'http:/127.0.0.1:5000/test/seccall'
response = requests.get(url)
response.raise_for_status()
responseJson = response.json()
if self.track:
print 'Response:::%s' %str(responseJson)
return responseJson
When I hit http://127.0.0.1:5000/test/seccall, I get the expected output. When I hit /test/triggerSecCall, the server stop responding. The request waits forever.
At this stage, I am not able to access any apis from anyother REST clients. When I force stop the server(Ctrl+C) I am getting response in the second REST client.
Why flask is not able to serve to internal service call?
I guess you are using the single threaded development server and not a WSGI setup for production.
Since the server has only one thread is can handle one request at a time. The first request will be executed, resulting in the requests.get(...) which will open a second request that can not be handled until the first request is complete, a dead lock.
The best solution would be to just call triggerSub() to get the result instead of using an HTTP request.
I am making a drop box like service using openstack. I am making a web interface using flask. User gets the object data in content of get request. I am sending data to the user iteratively. But my Flask app stops until the whole object is dowloaded. How could I make it non blocking?
#Returns the json content
r = swift_account.getObject(container_name, object_name)
filename = r.headers['X-Object-Meta-Orig-Filename']
#Make a generator so that all the content are not stored at once in memory
def generate():
for chunk in r.iter_content():
yield chunk
response = make_response(Response(stream_with_context(generate())))
response.headers['Content-Disposition'] = 'attachment; filename=' + filename
return response
Whether flask runs as blocking or non-blocking depends how you run it. When you say it blocks, how are you running it?
Unless it's being run by something with async capabilities like gunicorn with async or at least a threading model for multiple requests like apache with mod_wsgi then it won't be able to respond to more than one request at a time.