i have a node server and a django server. I try to send large file from the node to the django server.
I send the large file from node server by using fs.createReadStream() function. Each chunk created is send to the django server.
I'am new in Django, so i don't know how to handle chunks. I tried to look the doc, but nothing seems clear for me.
I need to store entire file in MEDIA_ROOT path.
var output = fs.createWriteStream("../tmp_upload/" + pathname.split('/')[0] + ".zip");
archive.pipe(output);
archive
.directory('../tmp_upload/' + pathname.split('/')[0])
.finalize();
output.on('close', function(){
const stats = fs.statSync("../tmp_upload/" + pathname.split('/')[0] + ".zip")
const fileSizeInBytes = stats.size
let content = fs.createReadStream("../tmp_upload/" + pathname.split('/')[0] + ".zip")
var chunk_sum = 0
var index = 0
content.on('data', chunk => {
chunk_sum += chunk.length
const data = {
token: req.session.userToken,
patientfile: {
file: chunk.toString(),
sumOfChunks: chunk_sum,
fileSize: fileSizeInBytes,
index : index,
label: pathname.split('/')[0] + ".zip",
category: "OT",
}
};
index += 1
Store.patientFileAdd(data) // here is where i send the chunk to django
});
here is a part of my nodejs function
class PatientFileAdd(views.APIView):
permission_classes = (AllowAny, )
def post(self, request, *args, **kwargs):
try:
filedata = request.data['patientfile']
file = filedata['file']
file_start = filedata['index']
upload_folder = "/patient/patient_%s" % request.userprofile.patientaccount, filedata['label']
with open(upload_folder, 'wb+') as destination:
for chunk in file.chunks():
destination.write(chunk)
return toolsViews.ResponseHandler(
status.HTTP_202_ACCEPTED,
{
'patientfile': 'OK'
}
)
Here my Django view
Thanks
Related
When I try to send image from flutter to flask, flask shows error 400.
I have no idea where is an error in my flutter code. Flutter function gets file(image in my case) as Uint8List. Then, I cast it as List, and trying to send with multipart.
Here is the code from flask and flutter.
Flask:
#auth.post('update/avatar')
#jwt_required()
def update_avatar():
current_user = get_jwt_identity()
save_folder = 'images/users/'
file = request.files.get('file', None)
file.filename = str(current_user) +".jpeg"
filename = secure_filename(file.filename)
file.save(os.path.join(save_folder, filename))
Flutter:
Future<String> uploadAvatar(Uint8List file, int userId) async {
var url = ApiConstants.baseUrlAuth + ApiConstants.updateAvatar + userId.toString();
String? access = await storage.storage.read(key: 'access');
if(access == null){
return '';
}
http.MultipartRequest request = http.MultipartRequest('POST', Uri.parse(url));
List<int> _selectedFile = file;
request.headers.addAll({'Authorization': access, "Content-type": "multipart/form-data"});
request.files.add(http.MultipartFile.fromBytes('file', _selectedFile, contentType: MediaType('file', 'jpeg'),));
http.StreamedResponse response = await request.send();
final responseStr = await response.stream.bytesToString();
Map data = json.decode(responseStr);
if (response.statusCode == 401 && data.containsKey("msg") && data['msg'] == "Token has expired!"){
String res = auths.refreshToken() as String;
if(res == "success"){
res = uploadImagePost(file, userId) as String;
}
return res;
} else if(response.statusCode == 201){
return data['photo_url'];
}
return '';
}
}
According to http Error 400 (Bad Request) the error is because request was somehow corrupted on the way.
check this Mozilla docs for more information about the main error.
I've created an angular component (product) on which a user is able to change an image (product's image).
html:
<img [attr.id]="_product.id" [attr.src]="_product.thumbnail_url"/>
When the image is chosen by the user from a custom file picker component on a modal form. The following method is called to reload the image in the parent product component.
The imageBinary here is a base64 blob binary image returned from a python FLASK REST API endpoint which is connected to a mysql database.
reloadProductImage(new_image_id):void{
let newUrl:SafeUrl;
let urlStr:string;
this.productservice.getImage(new_image_id).subscribe((filedata) =>{
let imageBinary = filedata.thumbnail;
urlStr = 'data:image/jpeg;base64,' + imageBinary + '?'+Date.now();
newUrl = this.domSanitizer.bypassSecurityTrustUrl(urlStr);
this.form.patchValue({image_id: new_image_id});
this._product.thumbnail_url = newUrl;
this._product.image_id = new_image_id;
I've found many examples how to force the browser to a reload on a regular src url by appending a timestamp to the url. However, these examples do not seem to apply to a blob stored in a saferurl.
inspect image element
<img _ngcontent-snv-c187="" id="146914e3-6e0a-409f-8046-ff04580f6242" src="data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgA+++very+long+blob+++vBzu3ZFwwAAAABJRU5ErkJggg==?1619156455998">
Console Error message
Failed to load resource: net::ERR_INVALID_URL
Failed attempts
I've tried:
newUrl = 'data:image/jpeg;base64,' + imageBinary + '?'+ Date.now();
and
newUrl = 'data:image/jpeg;base64,' + imageBinary + '?datetime='+ Date.now();
Background info
For illustration purposes, I'll post my Angular and Python code here (methods in a Flask SQLAlchemy DB Model) which post an image file to a Flask REST API, generate the thumbnail from the base64 image and return it to Angular. Note: the get function is called in a GET request. The Create function is called in a POST request.
Angular FileUpload component:
import { HttpClient } from '#angular/common/http';
import { Component, OnInit, Output, EventEmitter } from '#angular/core';
import {NotificationService} from '../..//notification/notification.service';
#Component({
selector: 'fileupload-form',
template: `
<p>
Productfoto toevoegen
</p>
<input type="file" (change)="onFileSelected($event)">
<button type="button" (click)="onUpload()">Upload</button>
`,
styles: [
]
})
export class FileuploadComponent implements OnInit {
selectedFile: File = null;
#Output()
fileSubmitted = new EventEmitter();
constructor(private http: HttpClient,
private notificationService: NotificationService,
) { }
onFileSelected(event){
this.selectedFile = <File>event.target.files[0];
}
onUpload(){
const fd = new FormData();
fd.append('inputFile', this.selectedFile , this.selectedFile.name);
this.http.post(['/offerte/api/files','upload'].join('/'),fd)
.subscribe(res => {
console.log(res);
this.notificationService.success(`Bestand ${this.selectedFile.name} is opgeslagen.`);
this.fileSubmitted.emit(res);
})
}
ngOnInit(): void {
}
}
Python Flask Endpoints:
Imports
import logging
from io import BytesIO
from flask import request, send_file
from flask_restx import Resource
from api.serializers import fileobject, page_of_files
from api.parsers import pagination_arguments
from api.restplus import api
from database.models import File
from flask_jwt_extended import jwt_required, get_jwt_identity
logger = logging.getLogger()
ns_files = api.namespace('files', description='Operations related to file management')
POST
#ns_files.route('/upload')
class Upload(Resource):
#api.doc(security='jwt')
#jwt_required
def post(self):
'''Upload a new file, returns id of new file'''
inputFile = request.files['inputFile']
logger.info('create new file %s', request.json)
current_user = get_jwt_identity()
return File.create(inputFile, current_user)
GET
#ns_files.route('/<string:file_id>')
#api.response(404, 'file not found')
class Details(Resource):
#api.marshal_with(fileobject)
def get(self, file_id):
'''Download a new file from the db (binary storage)'''
logger.info('loading file %s')
current_user = get_jwt_identity()
returnedFile = File.get(file_id)
return returnedFile
Python FLASK REST API functions
from PIL import Image
import magic
from base64 import b64encode
import io
accepted_mime_types = {'jpe':'image/jpeg','jpeg':'image/pjpeg','png':'image/png'}
def get(id):
return Product.query.filter(Product.id == id).one()
def create(inputFile, current_user):
filename = inputFile.filename
file_obj = File(
filename = filename,
extension = '.jpg',
folderpath = 'folderpath',
databinary = inputFile.read(),
category = 'default',
description = 'beschrijving...',
original_filename = filename,
created_by = current_user,
modified_by= current_user)
MAX_SIZE = (150, 150)
thumbnailFile, mimetype = File.getThumbnail(inputFile, MAX_SIZE)
if not thumbnailFile is None:
output = io.BytesIO()
thumbnailFile.save(output, format='PNG')
hex_data = output.getvalue()
file_obj.thumbnail = hex_data
else:
raise NameError(mimetype)
db.session.add(file_obj)
db.session.commit()
return str(file_obj.id)
def mimetype(file_obj):
return magic.from_buffer(file_obj.read(2048), mime=True)
def getThumbnail(file_obj, size):
file_obj.seek(0)
real_mime_type = magic.from_buffer(file_obj.read(2048), mime=True)
if (real_mime_type in accepted_mime_types.values()):
thumbnailImage = Image.open(file_obj)
thumbnailImage.thumbnail(size)
return thumbnailImage, real_mime_type
else:
raise NameError(real_mime_type)
I've build a small flask application with a file input form, and now I want that file to be sent to an API.
#app.route('/test', methods=['POST', 'GET'])
def test():
if request.method == 'POST':
file = request.files['file']
file.save(secure_filename(file.filename))
My Layout for the post part:
data = {
"local": file,
"name": file.filename
}
rp = requests.post(f'https://www.meistertask.com/api/tasks/{task_id}/attachments', data, headers={'Authorization': f'Bearer {access_token}'})
print(rp.status_code)
print(rp.content)
No matter how I try to post that file to the API, I always get this response:
{
"errors": [
{
"message": "Parameter local should be of type ActionDispatch::Http::UploadedFile!",
"status": 400
}
]
}
I really don't know how to post a file of that type from my uploaded file in flask.
I'd really appreciate any help! Thank you very much in advance.
file is not a real file, but a filestorage from the werkzeug package.
https://werkzeug.palletsprojects.com/en/1.0.x/datastructures/#werkzeug.datastructures.FileStorage
The API documentation of meistertask is not clear to me, especially I cannot see what local should be. A path? A stream? Bytes?
I got it working with python urlib... maybe this helps you
f = open(fname,'rb')
content = f.read()
f.close()
url = 'https://www.meistertask.com/api/tasks/%s/attachments' % (task_id)
request = urllib.request.Request("%s" %(url))
boundary = "5645645645654"
request.add_header("User-Agent", "python-test")
request.add_header("Accept", "*/*")
request.add_header("Accept-Encoding", "gzip, deflate")
request.add_header("Content-Type","multipart/form-data; boundary=%s" % boundary)
request.add_header("Connection", "keep-alive")
request.add_header("Expect", "100-continue")
request.add_header("Authorization", "Bearer 11111111111111111111")
body = bytes("--%s\r\n" % boundary,'utf-8')
body += bytes('Content-Disposition: form-data; name="name"\r\n\r\n','utf-8')
body += bytes('%s\r\n' % fname,'utf-8')
body += bytes("--%s\r\n" % boundary,'utf-8')
body += bytes('Content-Disposition: file; name="local"; filename="%s"\r\n' % fname,'utf-8')
body += bytes('Content-Type: text/plain\r\n\r\n','utf-8')
body += content
body += bytes('\r\n','utf-8')
body += bytes("\r\n--%s--\r\n" % boundary,'utf-8')
request.data = body
#print(body)
f = urllib.request.urlopen(request)
response = f.read() #bytes
I need to let the Django auto download the generated file.
Tried all different solutions online, none of them works.
Views.py
def validate(request):
if request.method == 'POST':
filename = request.POST.get('source_file')
file_path = os.path.join(settings.MEDIA_ROOT, 'SourceFiles', filename)
region = request.POST.get('region')
product_type = request.POST.get('product_type')
result = validateSource.delay(file_path, region, product_type)
output_filepath, log_filepath = result.get()
if os.path.exists(output_filepath) and os.path.exists(log_filepath):
zip_filename = zipFiles([output_filepath, log_filepath], filename)
zip_filepath = os.path.join(settings.MEDIA_ROOT, zip_filename)
response = FileResponse(open(zip_filepath, 'rb'), as_attachment=True)
return response
raise Http404
Template: code for the form POST.
$(document).on('submit', '#productForm', function(e){
e.preventDefault();
var inputFilePath = document.getElementById('sourceFileInput').files.item(0).name;
$.ajax({
method: 'POST',
url: 'validate/',
data: {
source_file: inputFilePath,
region: $("#Region-choice").val(),
product_type: $("#Product-type").val()}
})
.done(function(){
document.getElementById('lblStatus').innerHTML = "Result: <br/>"
document.getElementById('lblStatusContent').innerHTML = "Success!"
})
.fail(function(req, textStatus, errorThrown) {
document.getElementById('lblStatus').innerHTML = "Result: <br/>"
alert("Something went wrong!:" + textStatus + ' ' + errorThrown )
});
});
});
It's not possible to download files to your computer via an ajax (XHR) request. So you need to redirect the user actually (setting window.location) to a view that downloads the file. Or you can add as a result of the successful POST a button the current page so the user can download the file. In any case, you need to move the file download to a different view so a standard GET request can fetch it.
But your code to return the file in Django (using FileResponse) is correct.
There's also an explanation with an alternative way of doing it here
def validate(request):
if request.method == 'POST':
filename = request.POST.get('source_file')
file_path = os.path.join(settings.MEDIA_ROOT, 'SourceFiles', filename)
region = request.POST.get('region')
product_type = request.POST.get('product_type')
result = validateSource.delay(file_path, region, product_type)
output_filepath, log_filepath = result.get()
if os.path.exists(output_filepath) and os.path.exists(log_filepath):
zip_filename = zipFiles([output_filepath, log_filepath], filename)
zip_filepath = os.path.join(settings.MEDIA_ROOT, zip_filename)
with open(zip_filepath, 'rb') as fh:
response = HttpResponse(fh.read(), content_type="application/force-download")
response['Content-Disposition'] = 'attachment; filename=' + os.path.basename(zip_filepath)
return response
raise Http404
I'm using cytoscape.js to create a menu where I can download files quickly. I'm using an Ajax POST in order to pass the file name back to flask in order to download. For some reason, I can get all the information back but for whatever reason I cannot get the file to download. I've tried two methods so far.
AJAX Post:
{{
content: 'Download',
select: function(ele) {{
//this is to get the name of URI
var loc = window.location.pathname
var postData = {{
"element": ele.id(),
"source": loc
}}
$.ajax({{
url: '/get_file',
type: "POST",
contentType: 'application/json',
data: JSON.stringify(postData),
dataType: 'json',
success: function(response) {{
console.log("got it!")
}},
error: function(xhr) {{
console.log("Nope!")
}}
}})
Now for the flask back end, we have method one, and the commented section shows method 2 & 3 (in the if loop (if os.path.isfile(SAVE_PATH)):
#app.route('/get_file', methods=['POST'])
def get_file():
print("This is request data: {}".format(request.data))
requests = request.get_json()
element = requests['element']
source = requests['source']
#this is where loc is retrieved from ajax post, in format of /static/{filename}.html
for change in ['/static/', '.html']:
if change in source:
source = source.replace(change,"")
print("source: {}".format(source))
SAVE_PATH = os.path.curdir + "/results/" + source + "/" + element
SAVE_DIRECTORY = os.path.curdir + "/results/" + source + "/"
if os.path.isfile(SAVE_PATH):
downloaded_file = open("{}".format(SAVE_PATH), 'rb').read()
#res = send_from_directory(SAVE_PATH.replace("./", ""), element, attachment_filename=element, mimetype="application/octet-stream", as_attachment=True)
#res = send_file(SAVE_PATH, as_attachment=True, attachment_filename=element, mimetype='application/octet-stream')
#return res
return Response(
downloaded_file,
mimetype="application/octet-stream",
headers={"Content-disposition":
"attachment; filename={}".format(element)})
else:
print("failed")
return "failed"
Now I'm getting all the correct response, when I print out downloaded_file, I get the binary output, but for whatever reason, it just isn't downloading.
Looks like you could construct the download path on the frontend and request the file directly, something like this:
location = loc.split("/");
path = "results/" + location[location.length - 1].split(".")[0] + "/" + ele.id();
window.location.href = path;