I have a Flask application that shall provide an endpoint to download a large file. However, instead of providing it from the file system or generating the file on-the-fly, this file has to be downloaded first from another server via HTTP.
Of course, I could perform a GET request to the external server first, download the file completely and store it in the file system or in memory and then as a second step provide it as a result for the original request. This would look for example like this (also including a basic authentication to indicate why a simple proxy on a lower layer is not sufficient):
#!flask/bin/python
from flask import Flask, jsonify
import os
import requests
from requests.auth import HTTPBasicAuth
app = Flask(__name__)
#app.route('/download')
def download():
auth = HTTPBasicAuth("some_user", "some_password")
session = requests.Session()
session.auth = auth
response = session.get("http://example.com")
return response.content
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1234, debug=True)
However, this increases both the latency and the storage requirements of the application. And also, even if the receiver only requires to perform a partial download (i.e. it performs a HTTP range request) of the file, it has to be fetched from the external server completely, first.
Is there a more elegant option to solve this, i.e. to provide support for HTTP range requests that are directly forwarded to the external server?
According to Proxying to another web service with Flask, Download large file in python with requests and Flask large file download I managed to make a Flask HTTP proxy in stream mode.
from flask import Flask, request, Response
import requests
PROXY_URL = 'http://ipv4.download.thinkbroadband.com/'
def download_file(streamable):
with streamable as stream:
stream.raise_for_status()
for chunk in stream.iter_content(chunk_size=8192):
yield chunk
def _proxy(*args, **kwargs):
resp = requests.request(
method=request.method,
url=request.url.replace(request.host_url, PROXY_URL),
headers={key: value for (key, value) in request.headers if key != 'Host'},
data=request.get_data(),
cookies=request.cookies,
allow_redirects=False,
stream=True)
excluded_headers = ['content-encoding', 'content-length', 'transfer-encoding', 'connection']
headers = [(name, value) for (name, value) in resp.raw.headers.items()
if name.lower() not in excluded_headers]
return Response(download_file(resp), resp.status_code, headers)
app = Flask(__name__)
#app.route('/', defaults={'path': ''})
#app.route('/<path:path>')
def download(path):
return _proxy()
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1234, debug=True)
download_file() will open the request in stream mode and yield every chunk as soon as they got streamed.
_proxy() create the request then just create and return a Flask Response using the iterator download_file() as content.
I tested it with https://www.thinkbroadband.com/download where several archive files are free to download for test purpose. (be careful, archives are corrupted, so you better use checksum to make sure you got the expected file).
Some examples:
curl 'http://0.0.0.0:1234/100MB.zip' --output /tmp/100MB.zip
curl 'http://0.0.0.0:1234/20MB.zip' --output /tmp/20MB.zip
I also performed some other tests on random websites to get large images. So far I got no issues.
Related
I have a purest suite for my flask app that works great. However, I want to test some of my code that uses a third-party library (Qt) to send http requests. How is this possible? I see flask-testing has the live_server fixture which accomplishes this along with flask.url_for(), but it takes too much time to start up the server in the fixture.
Is there a faster way to send an http request from a third-party http lib to a flask app?
Thanks!
Turns out you can do this by manually converting the third-party request to the FlaskClient request, using a monkeypatch for whatever "send" method the third-party lib uses, then convert the flask.Response response back to a third-party reply object. All this occurs without using a TCP port.
Here is the fixture I wrote to bridge Qt http requests to the flask app:
#pytest.fixture
def qnam(qApp, client, monkeypatch):
def sendCustomRequest(request, verb, data):
# Qt -> Flask
headers = []
for name in request.rawHeaderList():
key = bytes(name).decode('utf-8')
value = bytes(request.rawHeader(name)).decode('utf-8')
headers.append((key, value))
query_string = None
if request.url().hasQuery():
query_string = request.url().query()
# method = request.attribute(QNetworkRequest.CustomVerbAttribute).decode('utf-8')
# send
response = FlaskClient.open(client,
request.url().path(),
method=verb.decode('utf-8'),
headers=headers,
data=data)
# Flask -> Qt
class NetworkReply(QNetworkReply):
def abort(self):
pass
reply = NetworkReply()
reply.setAttribute(QNetworkRequest.HttpStatusCodeAttribute, response.status_code)
for key, value in response.headers:
reply.setRawHeader(key.encode('utf-8'), value.encode('utf-8'))
reply.open(QIODevice.ReadWrite)
reply.write(response.data)
QTimer.singleShot(10, reply.finished.emit) # after return
return reply
qnam = QNetworkAccessManager.instance() # or wherever you get your instance
monkeypatch.setattr(qnam, 'sendCustomRequest', sendCustomRequest)
return ret
I have a HTTP Server in Python which is able to take and save files sent via POST request. However, now I am trying to download requested files with GET request. Basically, I want to send a GET request with a filename and I want my server to offer me a file to download. How would I do so?
My code so far:
class S(BaseHTTPRequestHandler):
def _set_headers(self):
self.send_response(200)
self.send_header("Content-type", "text/html")
self.end_headers()
def _html(self, message):
"""This just generates an HTML document that includes `message`
in the body. Override, or re-write this do do more interesting stuff.
"""
content = f"<html><body><h1>{message}</h1></body></html>"
return content.encode("utf8") # NOTE: must return a bytes object!
def do_GET(self):
self._set_headers()
self.wfile.write(self._html("hi!"))
def do_HEAD(self):
self._set_headers()
def do_POST(self):
form = cgi.FieldStorage(
fp=self.rfile,
headers=self.headers,
environ={'REQUEST_METHOD':'POST',
'CONTENT_TYPE':self.headers['Content-Type'],
})
filename = form['file'].filename
data = form['file'].file.read()
open(filename, "wb").write(data)
self._set_headers()
self.wfile.write(self._html("hi!"))
Instead of writing custom methods for http response and routing
engines you can use flask a lighwieght framwork that provide routing
engine and all features and security to the app and this also provide
necessary things to manage all kind of http requests
https://www.palletsprojects.com/p/flask/
Please maintain below structure in your root directory
app.py
from flask import Flask
app = Flask(__name__)
main.py
from flask_cors import CORS
import os
from app import app
# allowing cross browser request with cors configuration
CORS(app, supports_credentials=True,methods=['GET', 'HEAD', 'POST', 'OPTIONS', 'PUT', 'PATCH', 'DELETE'])
if __name__ == "__main__":
app.session_type = os.getenv('SESSION_TYPE')
app.run(debug=os.getenv('DEBUG'), host=os.getenv(
'SERVER_HOST'), port=os.getenv('SERVER_PORT'))
then execute the python main.py it will activate all routes enjoyy!!
the api should include one function called "write text to file" and inputs a string parameter
as for the function to write to the disk I have no problem and I implemented the code my problem is how to set the rest API using python.
EDIT:
this is my code:
from flask import (
Flask,
render_template
)
import SocketServer
import SimpleHTTPServer
import re
app = Flask(__name__, template_folder="templates")
#app.route('/index', methods=['GET'])
def index():
return 'Welcome'
#app.route('/write_text_to_file', methods=['POST'])
def write_text_to_file():
f = open("str.txt", "w+")
f.write("hello world")
f.close()
if __name__ == '__main__':
app.run(debug=True)
anyhow when I try to test my rest api:
http://127.0.0.1:5000/write_text_to_file
I am getting the following error:
Now I'm trying to test my rest-api , however how can I make my code to start the server and to the test the post request api, this is my test_class:
import requests
import unittest
API_ENDPOINT="http://127.0.0.1:5000/write_text_to_file"
class test_my_rest_api(unittest.TestCase):
def test_post_request(self):
"""start the server"""
r = requests.post(API_ENDPOINT)
res = r.text
print(res)
also when runnning my request using postman I am getting internal_server_error:
You're doing a GET request for this url, but you've specified that this endpoint can only accept POST:
#app.route('/write_text_to_file', methods=['POST'])
Also, the SocketServer and SimpleHTTPServer imports are not needed with Flask.
The method is not allowed because Chrome (or any browser) makes GET requests.
Whereas, you defined it as POST
#app.route('/write_text_to_file', methods=['POST'])
Either change it to a GET method, or use a tool such as POSTMan to perform other HTTP call types
I am new to API, and get a tasks of creating POST API. I have created a code somehow.
I want to add data to the hello.txt through post API, So how will I do it?
Here is my code:
import flask
from flask import request, jsonify
app = flask.Flask(__name__)
app.config["DEBUG"] = True
#app.route('/api/v1/resources/messages', methods = ['POST'])
def api_message():
if request.headers['Content-Type'] == 'text/plain':
return "Text Message: " + request.data
elif request.headers['Content-Type'] == 'application/octet-stream':
return "Binary message written!"
elif request.headers['Content-Type'] == 'application/json':
f = open('F:\Asif_Ahmed\Projects\api\hello.txt',"w")
f.write(request.data)
f.close()
return "JSON Message: " + json.dumps(request.json)
else:
return "415 Unsupported Media Type ;)"
app.run()
from flask import Flask, jsonify, render_template, request #import flask library
from flask_basicauth import BasicAuth # import flask library for create basic authentication if needed
from flask_cors import CORS # import flask library Cross-Origin Resource Sharing that is a mechanism that uses additional HTTP headers to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin
app = Flask(__name__)
CORS(app) #set-up cors for my app
#if you want use basic authentication you need set-up username and password
app.config['BASIC_AUTH_USERNAME'] = 'admin'
app.config['BASIC_AUTH_PASSWORD'] = 'password'
basic_auth = BasicAuth(app)#set-up username and password for my app but in this case I'm not specifying yet in which API use them
#app.route('/api/v1/resources/add_messages', methods=['POST'])#create my POST api
#basic_auth.required# set-up basic authentication for this API, comment out if not needed
def update_credential ():
json_credential=request.get_json()#get the JSON sent via API
print (json_credential["message"])#get the node "message" of my JSON
###########
#code to write in your file, you need write the json_credential["message"]
###########
return ("ok")
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1024, threaded=True)#start my flask app with local_host IP and specific port, if you don't specify the port it will run in the default port
In this case the JSON Input should be:
{"message":"your text"}
Please let me know if something is not clear, I even try this code on my local and the JSON is passed without problems.....
So you need run your python script and see that the API is running, if you had no JSON to send and was just a simple API that give back information you should have used even Chrome but in this case that you need send some JSON data I would advice you to use Postman.
See screenshot example:
this is a two-part question: I have seen individual pieces discussed, but can't seem to get the recommended suggestions to work together. I want to create a web service to store images and their metadata passed from a caller and run a test call from Postman to make sure it is working. So to pass an image (Drew16.jpg) to the web service via Postman, it appears I need something like this:
For the web service, I have some python/flask code to read the request (one of many variations I have tried):
from flask import Flask, jsonify, request, render_template
from flask_restful import Resource, Api, reqparse
...
def post(self, name):
request_data = request.get_json()
userId = request_data['UserId']
type = request_data['ImageType']
image = request.files['Image']
Had no problem with the data portion and straight JSON but adding the image has been a bugger. Where am I going wrong on my Postman config? What is the actual set of Python commands for reading the metadata and the file from the post? TIA
Pardon the almost blog post. I am posting this because while you can find partial answers in various places, I haven't run across a complete post anywhere, which would have saved me a ton of time. The problem is you need both sides to the story in order to verify either.
So I want to send a request using Postman to a Python/Flask web service. It has to have an image along with some metadata.
Here are the settings for Postman (URL, Headers):
And Body:
Now on to the web service. Here is a bare bones service which will take the request, print the metadata and save the file:
from flask import Flask, request
app = Flask(__name__)
# POST - just get the image and metadata
#app.route('/RequestImageWithMetadata', methods=['POST'])
def post():
request_data = request.form['some_text']
print(request_data)
imagefile = request.files.get('imagefile', '')
imagefile.save('D:/temp/test_image.jpg')
return "OK", 200
app.run(port=5000)
Enjoy!
Make sure `request.files['Image'] contains the image you are sending and follow http://flask.pocoo.org/docs/1.0/patterns/fileuploads/ to save the file to your file system. Something like
file = request.files['Image']
file.save('./test_image.jpg')
might do what you want, while you will have to work out the details of how the file should be named and where it should be placed.