Develop Reference

FastAPI: How to download bytes through the API

Is there a way to download a file through FastAPI? The files we want are located in an Azure Datalake and retrieving them from the lake is not an issue, the problem occurs when we try to get the bytes we get from the datalake down to a local machine.
We have tried using different modules in FastAPI such as starlette.responses.FileResponse and fastapi.Response with no luck.
In Flask this is not an issue and can be done in the following manner:
from io import BytesIO
from flask import Flask
from werkzeug import FileWrapper
flask_app = Flask(__name__)
#flask_app.route('/downloadfile/<file_name>', methods=['GET'])
def get_the_file(file_name: str):
the_file = FileWrapper(BytesIO(download_file_from_directory(file_name)))
if the_file:
return Response(the_file, mimetype=file_name, direct_passthrough=True)
When running this with a valid file name the file automatically downloads. Is there equivalent way to this in FastAPI?
Solved
After some more troubleshooting I found a way to do this.
from fastapi import APIRouter, Response
router = APIRouter()
#router.get('/downloadfile/{file_name}', tags=['getSkynetDL'])
async def get_the_file(file_name: str):
# the_file object is raw bytes
the_file = download_file_from_directory(file_name)
if the_file:
return Response(the_file)
So after a lot of troubleshooting and hours of looking through documentation, this was all it took, simply returning the bytes as Response(the_file).

After some more troubleshooting I found a way to do this.
from fastapi import APIRouter, Response
router = APIRouter()
#router.get('/downloadfile/{file_name}', tags=['getSkynetDL'])
async def get_the_file(file_name: str):
# the_file object is raw bytes
the_file = download_file_from_directory(file_name)
if the_file:
return Response(the_file)
So after a lot of troubleshooting and hours of looking through documentation, this was all it took, simply returning the bytes as Response(the_file) with no extra parameters and no extra formatting for the raw bytes object.

As far as I know, you need to set media_type to the adequate type. I did that with some code a year ago and it worked fine.
#app.get("/img/{name}")
def read(name: str, access_token_cookie: str=Cookie(None)):
r = internal.get_data(name)
if r is None:
return RedirectResponse(url="/static/default.png")
else:
return Response(content=r["data"], media_type=r["mime"])
r is a dictionary with the data as raw bytes and mime the type of the data as given by PythonMagick.

To add a custom filename to #Markus's answer, in case your api's path doesn't end with a neat filename or you want to determine a custom filename from server side and give to the user:
from fastapi import APIRouter, Response
router = APIRouter()
#router.get('/downloadfile/{file_name}', tags=['getSkynetDL'])
async def get_the_file(file_name: str):
# the_file object is raw bytes
the_file = download_file_from_directory(file_name)
filename1 = make_filename(file_name) # a custom filename
headers1 = {'Content-Disposition': f'attachment; filename="{filename1}"'}
if the_file:
return Response(the_file, headers=headers1)

Flask to read and write to same csv

I have a flask app that was created using a csv file. Using html forms, I give the user the option to input new data, and then I write changes to the csv (using a custom function) and download it. This downloads just fine and saves to my desktop. Is there any way to tweak the code to save it in the same project directory and overwrite the csv that serves the flask app? This way the app might update upon refresh. Thanks!
#app.route('/csv/')
def download_csv():
model_id=request.args['textid']
client_id = session['client_id']
# return response
df=recommender.update_history(client_id, model_id)
df= recommender.get_csv()
resp = make_response(df.to_csv(encoding='iso-8859-1',index=False))
resp.headers["Content-Disposition"] = "attachment; filename=export.csv"
resp.headers["Content-Type"] = "text/csv"
return resp

The comment above is correct. I didn't need the make_response function. Here's what worked:
#app.route('/csv/')
def download_csv():
model_id=request.args['textid']
client_id = session['client_id']
# return response
df=recommender.update_history(client_id, model_id)
df= recommender.get_csv()
path=r'data/file_name.csv'
resp = df.to_csv(path, encoding='iso-8859-1',index=False)
return render_template('index.html')

How can I send data from HDFS through Django backend to Angular 5 frontend?

I am downloading file from Hadoop to Django backend and storing the file using the code below:
import shutil
import requests
url = 'http://112.138.0.12:9870/webhdfs/v1/user/username/1.jpg?op=OPEN&user.name=username'
response = requests.get(url, stream=True)
with open('img.png', 'wb') as out_file:
shutil.copyfileobj(response.raw, out_file)
del response
I don't need to store the file in backend local system since I want to send this file to Angular 5 frontend where user will save this file in their local system. I'm getting the following error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position
0: invalid start byte.
Can someone suggest me what would be the right way to do download large files in a short time?
DJANGO:
views.py:
class DownloadFileView(GenericAPIView):
serializer_class = UserNameSerializer
def get(self, request):
key = request.META.get('HTTP_AUTHORIZATION').split()[1]
user_id = Token.objects.get(key=key).user_id
user_name = User.objects.get(id=user_id).username
response = download_files(user_name)
return Response(response)
def download_files(user_name):
response = requests.get('http://112.138.0.12:9870/webhdfs/v1/user/' + user_name + '/1.jpg?op=OPEN&user.name=username', stream=True)
return response.raw
ANGULAR:
DownloadFile(){
this.userService.DownloadFiles().subscribe((data : any) => {
const blob = new Blob([data], { type: 'application/octet-stream'});
fileUrl = this.sanitizer.bypassSecurityTrustResourceUrl(window.URL.createObjectURL(blob));
}
}
DownloadFiles() {
this.token = localStorage.getItem('userToken')
var reqHeader = new HttpHeaders({ 'Content-Type': 'application/octet-stream', 'Authorization': 'token ' + this.token });
console.log(reqHeader)
return this.http.get(this.rootURL + 'download/', { headers: reqHeader});
}

To begin with your unicode error, it's because:
HttpResponse.init(content='', content_type=None, status=200,
reason=None, charset=None)
Instantiates an HttpResponse
object with the given page content and content type.
content should be an iterator or a string. If it’s an iterator, it
should return strings, and those strings will be joined together to
form the content of the response. If it is not an iterator or a
string, it will be converted to a string when accessed.
I do believe django is having trouble converting the binary data in the file to string. A more common approach when dealing with file downloads is:
response = HttpResponse(content_type="application/jpeg")
response.write(binary_data)
This works because there is a call to make_bytes behind the scenes which handles the binary data correctly.
Having said that, this is not the most efficient way to go about it. Your web app makes a request to a remote server using requests, and then passes that onto the client. Why not get your angular code to fetch the data directly from the end point?
Can't do that because you want authentication you say? Ok, How about checking the authentiation and then sending an HttpResponseDirect like this:
return HttpResponseRedirect('http://112.138.0.12:9870/webhdfs/v1/user/' + user_name + '/1.jpg?op=OPEN&user.name=username')

How to test file upload in Django rest framework using PUT?

I would like to write a unit test for a view on a Django REST Framework application. The test should upload a file using PUT, in essence equivalent to
http -a malkarouri PUT http://localhost:8000/data-packages/upload/ka #tmp/hello.py
The code I have written so far is
factory = APIRequestFactory()
request = factory.put( '/data-packages/upload/ka',
data,
content_type='application/octet-stream',
content_disposition="attachment; filename=data.dump")
force_authenticate(request, user)
view = PackageView.as_view()
response = view(request, "k.py")
which, obviously, does not upload a file. The specific error when running the test is 400:
{u'detail': u'Missing filename. Request should include a Content-Disposition header with a filename parameter.'}
Notably, I am using a request factory to test the view rather than a full client. That is what makes solutions such as the one in this question not work for me.
What is the correct way to set the content disposition header?

Hi you need to use the SimpleUploadedFile wrapper for that :
from django.core.files.uploadedfile import SimpleUploadedFile
from django.core.files import File
data = File(open('path/bond-data.dump', 'rb'))
upload_file = SimpleUploadedFile('data.dump', data.read(),content_type='multipart/form-data')
request = factory.put( '/data-packages/upload/ka',
{'file':upload_file,other_params},
content_type='application/octet-stream',
content_disposition="attachment; filename=data.dump")
Ps : I am using APITestCase

from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import TestCase
from rest_framework.test import APIClient
class MyTest(TestCase):
client_class = APIClient
def test_it(self):
file = SimpleUploadedFile("file.txt", b"abc", content_type="text/plain")
payload = {"file": file}
response = self.client.post("/some/api/path/", payload, format="multipart")
self.assertEqual(response.status_code, 201)
# If you do more calls in this method with the same file then seek to zero
file.seek(0)

How to speed up JSON for a flask application?

I'm currently implementing a webapp in flask. It's an app that does a visualization of data gathered. Each page or section will always have a GET call and each call will return a JSON response which then will be processed into displayed data.
The current problem is that some calculation is needed before the function could return a JSON response. This causes some of the response to arrive slower than others and thus making the page loads a bit slow. How do I properly deal with this? I have read into caching in flask and wonder whether that is what the app need right now. I have also researched a bit into implementing a Redis-Queue. I'm not really sure which is the correct method.
Any help or insights would be appreciated. Thanks in advance

Here are some ideas:
If the source data that you use for your calculations is not likely to change often then you can run the calculations once and save the results. Then you can serve the results directly for as long as the source data remains the same.
You can save the results back to your database, or as you suggest, you can save them in a faster storage such as Redis. Based on your description I suspect the big performance gain will be in not doing calculations so often, the difference between storing in a regular database vs. Redis or similar is probably not significant in comparison.
If the data changes often then you will still need to do calculations frequently. For such a case an option that you have is to push the calculations to the client. Your Flask app can just return the source data in JSON format and then the browser can do the processing on the user's computer.
I hope this helps.

You can use
copy_current_request_context and Redis, Thread
It is helpful when you need long time to make JSON response.
The first request maybe slow, but next request will faster.
Example
from datetime import timedelta, datetime
from threading import Thread
from . import dbb, redis_client
from flask import Blueprint, request, jsonify, flash, after_this_request, copy_current_request_context, \
current_app, send_from_directory
from .models import Shop, Customers
def save_customer_json_to_redis(request):
response_json = {
"have_customer": False,
"status": False,
"anythingelse": None,
"message":"False, you have to check..."
}
#print(request.data)
headers = request.headers
Authorization = headers['Authorization']
token = Authorization.replace("Bearer", "")
phone = request.args.get('phone')
if phone is not None and phone != "":
print('token', token, "phone", phone)
now = datetime.utcnow() + timedelta(hours=7)
shop = Shop.query.filter(Shop.private_token == token, Shop.ended_date > now, Shop.shop_active == True).first()
customer = Customers.query.filter_by(shop_id=shop.id, phone=phone).first()
if customer:
redis_name = f'{shop.id}_api_v2_customer_phone_{phone}_customer_id_{customer.id}'
print(redis_name)
response_json["anythingelse"] = ...# do want you want, it need long time to do
response_json["status"] = True
response_json["message"] = "Successful"
redis_client.set(redis_name, json.dumps(response_json)) #Save JSON to Redis
#app.route('/api/v2/customer', methods=['GET'])
def api_customer():
#copy_current_request_context
def do_update_customer_to_redis():# this function to save JSON you want to response next time to Redis
save_customer_json_to_redis(request)
Thread(target=do_update_customer_to_redis).start()
response_json = {
"have_customer": False,
"status": False,
"anythingelse": {},
"message": "False, you have to check..."
}
#print(request.data)
headers = request.headers
Authorization = headers['Authorization']
token = Authorization.replace("Bearer", "")
phone = request.args.get('phone')
if phone is not None and phone != "":
print('token', token, "phone", phone)
now = datetime.utcnow() + timedelta(hours=7)
shop = Shop.query.filter(Shop.private_token == token, Shop.ended_date > now,Shop.shop_active == True).first()
customer = Customers.query.filter_by(shop_id=shop.id, phone=phone).first()
if customer:
redis_name = f'{shop.id}_api_v2_customer_phone_{phone}_customer_id_{customer.id}'
print(redis_name)
try:
response_json = json.loads(redis_client.get(redis_name)) # if have json from app
print("json.loads(redis_client.get(redis_name))")
except Exception as e:
print("json.loads(redis_client.get(redis_name))", e)
#do any thing you want to response json
response_json["anythingelse"] = ...# do want you want, it need long time to do
response_json["message"]= ...#do want you want
#redis_client.set(redis_name, json.dumps(response_json))
response_json["status"] = True
response_json["message"] = "Successful"
return jsonify(response_json)
In the init.py
from flask import Flask
from flask_cors import CORS
from flask_mail import Mail
from flask_sqlalchemy import SQLAlchemy
from redis import Redis
# init SQLAlchemy so we can use it later in our models
dbb = SQLAlchemy(session_options={"autoflush": False})
redis_client = Redis(
host='localhost',
port='6379',
password='your_redis_password'
)
def create_app():
app = Flask(__name__)
...........