I'm sending a excel file from Angular to Django. I want to read the file using Pandas and perform some operations in the file, but I'm not sure how to do it.
class fileupload(APIView) :
def post(self, request):
f = request.FILES
print(f)
When I print, it shows below,
<MultiValueDict: {'excelfile': [<InMemoryUploadedFile: New_Excel.xlsx (application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)>]}>
Here, I want to save this file to some location and use pandas to perform operations or if possible, directly would need to read the file using pandas. I'm new to Django and Pandas so if anything is wrong, please help.. Thanks in advance
from django.core.files.storage import default_storage
from django.core.files.base import ContentFile
file_objs = request.data.getlist('files')
for file_obj in file_objs:
path = default_storage.save(settings.MEDIA_ROOT, ContentFile(file_obj.read()))
print("images path are",path)
class fileupload(APIView) :
def post(self, request):
f = request.data.getlist("excelfile")
print(f) # list of elements
Now loop the f then store one by one
Related
Trying to make endpoint that can read uploaded .sav (SPSS) file and create model with data from it. For getting data from it I'm using pyreadstat library. But now when I'm trying to run it I have an error expected str, bytes or os.PathLike object, not InMemoryUploadedFile
How I can change this code so pyreadstat can correctly read the given file?
from rest_framework import generics, status
import pandas as pd
import pyreadstat
from rest_framework.response import Response
from .models import Research, Data
from .serializers import FileUploadSerializer, SaveFileSerializer
class UploadFileView(generics.CreateAPIView):
serializer_class = FileUploadSerializer
def post(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
file = serializer.validated_data['file']
df, meta = pyreadstat.read_sav(file, user_missing=True)
json_df = df.to_json(orient='table')
rsch = Research.objects.get(pk=1)
Data.objects.create(research=rsch, data={})
Change this:
pyreadstat.read_sav(file, user_missing=True)
To this:
pyreadstat.read_sav(file.getvalue(), user_missing=True)
Hope it works.
Django uploaded files are mostly file like, and they only proxy certain methods. See the FileProxyMixin source for more details on specific methods, if you are curious.
To get the uploaded contents, use the read() method:
file = serializer.validated_data['file']
contents = file.read()
df, meta = pyreadstat.read_sav(contents, user_missing=True)
In this case you are getting an InMemoryUploadFile but this could change if Django decides to write to disk due to the upload being too large.
Documentation on django uploaded file objects:
https://docs.djangoproject.com/en/4.0/ref/files/uploads/#.
In my Django "sources" project I have the "bulletins" app.
I have a view "model_form_upload" inside of which I get the uploaded file name ("csvFilename") when I upload it.
views.py
from bulletins.forms import ErnageForm
from bulletins.models import ErnageModel
from bulletins.pretreatment import pretreatment
def model_form_upload(request):
if request.method == 'POST':
form = ErnageForm(request.POST, request.FILES)
if form.is_valid():
form.save()
for filename, file in request.FILES.items():
csvFilename = file.name
resultPretreatment = pretreatment()
print(resultPretreatment)
return redirect('proceed_FT')
else:
return redirect('upload_fail')
else:
form = ErnageForm()
return render(request, 'bulletins/upload_csv.html', {'form': form})
Beside that, I have a python script "pretreatment.py" that effectuates a bunch of pandas-dataframe transformations on my csv file.
pretreatment.py
def pretreatment(csvFileToTreat="..."
#... all the transformations with pandas dataframes...
df.to_csv(txtExportFilePath, index=False, header=None, sep='\t')
models.py
class ErnageModel(models.Model):
csv = models.FileField(upload_to="METEO ERNAGE/{0}/{1}".format(yearToTreat,monthToTreat))
My csvFilename will always be a string of this type : "Ernageyyyymm.csv".
My question here is : How can I collect my csvFilename variable out of the views.model_form_upload function to use its date information it in the pretreatment python script variables "csvFileToTreat" and "txtExportFilePath".
Those two ones are of the type "path/Ernage{0}{1}.format(yearToTreat,monthToTreat).
As I have to use it each month I would like to have my files uploaded in media/METEO ERNAGE/{0}/{1}".format(yearNumber,monthNumber) as you can see in the FileField upload_to and then everything happening in this directory. So the pretreatment charges the uploaded file and then export in it also.
I hope all this is as clear as possible, thank you for your help.
store csvFilename name in a session like this:
request.session['file']=csvFilename
and you can use this session where ever you want like this.
csvFilename =request.session['file']
I have a field where users upload an xlsx file In Django, however, im having trouble getting Django to return a simple text-based webpage from users uploading that xlsx file. Basically, I need Django to generate text from the cells of the xlsx file they uploaded. Im trying to use openpyxl to do this. How do I get Django to generate a new page with the results of the xlsx file after my script runs over it? Do I need to create a new template that redirects to it once the user submits the uploaded xlsx file? Here's an example.
# views.py
from django.shortcuts import render
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from .forms import UploadForm
import openpyxl
def index2(request):
if request.method != 'POST':
form = UploadForm()
else:
form = UploadForm(request.POST or None, request.FILES or None)
if form.is_valid():
form.save()
file = request.FILES['fileobj']
wb = openpyxl.load_workbook(file)
sheet = wb.get_sheet_by_name('Sheet1')
# script that runs overs the sheet here...
return HttpResponseRedirect(reverse('webpage with results'))
context = {'form': form}
return render(request, 'upload/upload_page.html', context)
I suggest decompose the problem. Right now you're trying to solve several things at once:
Upload the file
Save the file
Parse the file
Render a view of the file
I'd put 1) and 2) to the side and work on the XSLX loading/viewing part independently. For example, you could hard-code the path of the file in the view and get it rock solid that way. It's just development after all. Once it's good, remove the hard-coded file and connect up the real upload/save logic.
Realistically, you'll want to create a model class that represents say UploadedExcelFile. When the file is uploaded, create an instance and save it (i.e. a row in the database). Then you write a view that loads that model instance, which in turn can load the Excel file.
You might also want to consider ingesting the file rather than just storing on disk, or using some kind of async approach (e.g. django-q). This depends on a few factors, like what kind of information needs to be extracted from the file, is it editable etc. If it were me, and not knowing the bounds of your system, I'd do something like this:
Upload file
Store file in a temp area
Create a model object (i.e. database row) to represent the file and say the state is "processing".
Fire up a django-q job to ingest the file, giving it the filename
Return a view to this model.
The async job loads the file, parses the relevant aspects to json, stores the data in the model object (i.e. in the database), marks the model state as "ready".
The user can reload the page to see the data (or use websockets etc to tell it to reload).
I generate a file in python, and want to "upload" that file to the django database. This way it is automatically put inside the media folder, and organized neatly with all other files of my application.
Now here is what I tried: (type hinting used, since it's python 3.6)
# forms.py
class UploadForm(forms.ModelForm):
class Meta:
model = UploadedFile
fields = ('document',)
# models.py
class UploadedFile(models.Model):
document = models.FileField(upload_to=get_upload_path)
# mimetype is generated by filename on save
mimetype = models.CharField(max_length=255)
# ... additional fields like temporary
def get_upload_path(instance: UploadedFile, filename):
if instance.temporary:
return "uploaded_files/temp/" + filename
return "uploaded_files/" + filename
# views.py, file_out has been generated
with open(file_out, 'rb') as local_file:
from django.core.files import File
form = UploadForm(dict(), {'document': File(local_file)})
print(form.errors)
if form.is_valid():
file = form.save(commit=False)
# ... set additional fields
file.save()
form.save_m2m()
return file
Now this is not the only thing I've tried. First I've gone with setting the FileField directly, but that resulted in the save() to fail, while the mimetype field is set. Because the original file sits outside the media folder, and thus a suspicious file action is triggered.
Also, the form gives some feedback about the "upload", through the form.errors.
Depending on my approach, either the save() fails as mentioned above -- meaning the "uploading" does not actually copy the file in the media folder -- or the form returns the error that no file was transmitted, and tells to check the form protocol.
Now my theory is, that I would have to go and initialize my own instance of InMemoryUploadedFile, but I could not figure out how to do that myself, and no documentation was available on the internet.
It feels like I'm taking the wrong approach from the get go. How would one do this properly?
Do you have get_upload_path defined? If not, that would explain the errors you're getting.
From what I can see you're on the right track. If you don't need a dynamic path for your uploads, if you just want them in media/uploads, you can pass in a string value for upload_to (from the Django docs):
# file will be uploaded to MEDIA_ROOT/uploads
document = models.FileField(upload_to='uploads/')
First of all, thanks to Franey for pointing me at storage documentation which lead me to contentfile documentation.
The ContentFile actually solves the problem, because it basically is the self-instantiated version of InMemoryUploadedFile that I was looking for. It's a django File that is not stored on disk.
Here's the full solution:
# views.py, file_out has been generated
with open(file_out, 'rb') as local_file:
from django.core.files.base import ContentFile
# we need to provide a name. Otherwise the Storage.save
# method reveives a None-parameter and breaks.
form = UploadForm(dict(), {'document': ContentFile(local_file.read(), name=name)})
if form.is_valid():
file = form.save(commit=False)
# ... set additional fields
file.save()
form.save_m2m()
return file
Let's say I am selecting 10 files to be uploaded using django-filer. They initially have very random names. I'd like to have a set of rules according to which they ought to be renamed and only then passed for further processing (thumbnails etc.).
I need to actually rename everything, especially filename, not just Image model name.
I tried catching pre_save signal for Image model and altering instance.original_filename but that's not renaming a filename. Or maybe should I subclass and override something from filer package?
I'd be grateful for code example cause this is a little bit to hard for me.
I used form_valid(self, form) in views.py to process and manipulate my images. The complete code is a bit long and very specific, but I'll post a few snipplets which should show the idea of how to generate filenames:
def form_valid(self, form):
upload = self.request.FILES['profilbild_original'] #coming from a very simple form
self.request.user.student.profilbild_original = upload
self.request.user.student.save()
#no renaming was required here, but now I did some work:
inputfilepath = os.path.join(my_app.settings.MEDIA_ROOT, profilbild_path(self.request.user, str(upload)))
original = Image.open(inputfilepath)
original.thumbnail((200,200), Image.ANTIALIAS)
filename = str(upload)+'.thumbnail_200_200_aa.jpg'
filepath = profilbild_path(self.request.user, filename)
filepath = os.path.join(my_app.settings.MEDIA_ROOT, filepath)
original.save(filepath, 'JPEG', quality=90)
self.request.user.student.profilbild = profilbild_path(self.request.user, filename).replace("\\", "/")
self.request.user.student.save()
return super(ProfilbildView, self).form_valid(form)
profilbild_pathis a function according to https://docs.djangoproject.com/en/1.3/ref/models/fields/#django.db.models.FileField.upload_to :
def profilbild_path(instance, filename):
return os.path.join('profilbilder', str(instance.id), filename)
I hope this gives you some clues.