I have a purest suite for my flask app that works great. However, I want to test some of my code that uses a third-party library (Qt) to send http requests. How is this possible? I see flask-testing has the live_server fixture which accomplishes this along with flask.url_for(), but it takes too much time to start up the server in the fixture.
Is there a faster way to send an http request from a third-party http lib to a flask app?
Thanks!
Turns out you can do this by manually converting the third-party request to the FlaskClient request, using a monkeypatch for whatever "send" method the third-party lib uses, then convert the flask.Response response back to a third-party reply object. All this occurs without using a TCP port.
Here is the fixture I wrote to bridge Qt http requests to the flask app:
#pytest.fixture
def qnam(qApp, client, monkeypatch):
def sendCustomRequest(request, verb, data):
# Qt -> Flask
headers = []
for name in request.rawHeaderList():
key = bytes(name).decode('utf-8')
value = bytes(request.rawHeader(name)).decode('utf-8')
headers.append((key, value))
query_string = None
if request.url().hasQuery():
query_string = request.url().query()
# method = request.attribute(QNetworkRequest.CustomVerbAttribute).decode('utf-8')
# send
response = FlaskClient.open(client,
request.url().path(),
method=verb.decode('utf-8'),
headers=headers,
data=data)
# Flask -> Qt
class NetworkReply(QNetworkReply):
def abort(self):
pass
reply = NetworkReply()
reply.setAttribute(QNetworkRequest.HttpStatusCodeAttribute, response.status_code)
for key, value in response.headers:
reply.setRawHeader(key.encode('utf-8'), value.encode('utf-8'))
reply.open(QIODevice.ReadWrite)
reply.write(response.data)
QTimer.singleShot(10, reply.finished.emit) # after return
return reply
qnam = QNetworkAccessManager.instance() # or wherever you get your instance
monkeypatch.setattr(qnam, 'sendCustomRequest', sendCustomRequest)
return ret
Related
I have a Flask application that shall provide an endpoint to download a large file. However, instead of providing it from the file system or generating the file on-the-fly, this file has to be downloaded first from another server via HTTP.
Of course, I could perform a GET request to the external server first, download the file completely and store it in the file system or in memory and then as a second step provide it as a result for the original request. This would look for example like this (also including a basic authentication to indicate why a simple proxy on a lower layer is not sufficient):
#!flask/bin/python
from flask import Flask, jsonify
import os
import requests
from requests.auth import HTTPBasicAuth
app = Flask(__name__)
#app.route('/download')
def download():
auth = HTTPBasicAuth("some_user", "some_password")
session = requests.Session()
session.auth = auth
response = session.get("http://example.com")
return response.content
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1234, debug=True)
However, this increases both the latency and the storage requirements of the application. And also, even if the receiver only requires to perform a partial download (i.e. it performs a HTTP range request) of the file, it has to be fetched from the external server completely, first.
Is there a more elegant option to solve this, i.e. to provide support for HTTP range requests that are directly forwarded to the external server?
According to Proxying to another web service with Flask, Download large file in python with requests and Flask large file download I managed to make a Flask HTTP proxy in stream mode.
from flask import Flask, request, Response
import requests
PROXY_URL = 'http://ipv4.download.thinkbroadband.com/'
def download_file(streamable):
with streamable as stream:
stream.raise_for_status()
for chunk in stream.iter_content(chunk_size=8192):
yield chunk
def _proxy(*args, **kwargs):
resp = requests.request(
method=request.method,
url=request.url.replace(request.host_url, PROXY_URL),
headers={key: value for (key, value) in request.headers if key != 'Host'},
data=request.get_data(),
cookies=request.cookies,
allow_redirects=False,
stream=True)
excluded_headers = ['content-encoding', 'content-length', 'transfer-encoding', 'connection']
headers = [(name, value) for (name, value) in resp.raw.headers.items()
if name.lower() not in excluded_headers]
return Response(download_file(resp), resp.status_code, headers)
app = Flask(__name__)
#app.route('/', defaults={'path': ''})
#app.route('/<path:path>')
def download(path):
return _proxy()
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1234, debug=True)
download_file() will open the request in stream mode and yield every chunk as soon as they got streamed.
_proxy() create the request then just create and return a Flask Response using the iterator download_file() as content.
I tested it with https://www.thinkbroadband.com/download where several archive files are free to download for test purpose. (be careful, archives are corrupted, so you better use checksum to make sure you got the expected file).
Some examples:
curl 'http://0.0.0.0:1234/100MB.zip' --output /tmp/100MB.zip
curl 'http://0.0.0.0:1234/20MB.zip' --output /tmp/20MB.zip
I also performed some other tests on random websites to get large images. So far I got no issues.
I am working on an application which is based on GAE with python 2.7.13. What I want to do is that to make a bunch of async API calls inside a handler. Something like that:
class MakeRequests(webapp2.RequestHandler):
def post(self, *v, **kv):
*do an async api call#1*
*do an async api call#2*
*do an async api call#3*
*wait for response from all of above api requests*
*make response in a way like if call#1 failes, make it's expected*
*attributes in response as None, if call#2 succeeds add it's*
*attributes in response etc. This is just an example.*
For that purpose, I have tried libraries like asyncio, grequests, requests and simple-requests, they don't seems to be working because either they are not compatible with with GAE or with python 2.7.13.
Can anyone help me here?
Urlfetch, which is bundled by default with GAE has a way of making asynchronous calls:
from google.appengine.api import urlfetch
def post(self, *v, **kv):
rpcs = []
for url in urls:
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, url)
rpcs.append(rpc)
results = [rpc.get_result() for rpc in rpcs]
# do stuff with results
If, for some reason you don't want to use urlfetch you can parallelize the requests manually by using threading and a synchronized Queue to read the results.
I'm trying to write an HTTP server in python 2.7. I'm trying to use ready-made classes to simplify the job (such as SimpleHTTPServer, BaseHTTPRequestHandler, etc.).
The server should listen for GET requests, and once it gets one - parse the request (the path and the arguments), and interact with an already initialized object (which accesses a DB, counts number of requests, etc.) - let's call it the 'handler', and return a response.
I understand that the RequestHandler class (e.g. BaseHTTPRequestHandler) will be constructed for each request. How can I pass the 'handler' to the handling routines, so that they could call its methods?
Thanks!
Use a framework to further simplify your job. Here is an example in flask:
from flask import Flask
from flask import request
app = Flask(__name__)
your_handler = SomeHandlerClass()
#app.route("/")
def index():
return your_handler.do_something_with(request)
if __name__ == "__main__":
app.run()
request is a proxy object that holds all the incoming request data.
I am using the Flask XML-RPC extension and everything is working well. Now I want to protect the XML-RPC endpoint with a basic HTTP authentication using Flask HTTPAuth extension.
This extension is usually used with routes, but the XML-RPC endpoint is not defined as a route:
handler = XMLRPCHandler('xmlrpc')
handler.connect(app, '/xml-rpc')
def hello_word():
return "Hello"
handler.register_function(hello_world)
How can I use HTTP authentication with Flask-XML-RPC so that any caller of /xml-rpc has to authenticate?
You'll have to subclass the XMLRPCHandler() class; each call through /xml-rpc is handled by the XMLRPCHandler.handle_request() method, you can decorate that to handle authentication:
class HTTPAuthXMLRPCHandler(XMLRPCHandler):
#auth.login_required
def handle_request(self):
return XMLRPCHandler.handle_request(self)
handler = HTTPAuthXMLRPCHandler('xmlrpc')
handler.connect(app, '/xml-rpc')
I'm writing a REST client and I need to mock a HTTP server in my tests. What would be the most appropriate library to do that? It would be great if I could create expected HTTP requests and compare them to actual.
Try HTTPretty, a HTTP client mock library for Python helps you focus on the client side.
You can also create a small mock server on your own.
I am using a small web server called Flask.
import flask
app = flask.Flask(__name__)
def callback():
return flask.jsonify(list())
app.add_url_rule("users", view_func=callback)
app.run()
This will spawn a server under http://localhost:5000/users executing the callback function.
I created a gist to provide a working example with shutdown mechanism etc.
https://gist.github.com/eruvanos/f6f62edb368a20aaa880e12976620db8
You can do this without using any external library by just running a temporary HTTP server.
For example mocking a https://api.ipify.org?format=json
"""Unit tests for ipify"""
import http.server
import threading
import unittest
import urllib.request
class MockIpifyHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
"""HTTPServer mock request handler"""
def do_GET(self): # pylint: disable=invalid-name
"""Handle GET requests"""
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(b'{"ip":"1.2.3.45"}')
def log_request(self, code=None, size=None):
"""Don't log anything"""
class UnitTests(unittest.TestCase):
"""Unit tests for urlopen"""
def test_urlopen(self):
"""Test urlopen ipify"""
server = http.server.ThreadingHTTPServer(
("127.0.0.127", 9999), MockIpifyHTTPRequestHandler
)
with server:
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
request = request = urllib.request.Request("http://127.0.0.127:9999/")
with urllib.request.urlopen(request) as response:
result = response.read()
server.shutdown()
self.assertEqual(result, b'{"ip":"1.2.3.45"}')
Alternative solution I found is in https://stackoverflow.com/a/34929900/15862
Mockintosh seems like another option.