I'm using Django to host a web application, and I would like to obtain a CSV file from the user, and process this information (using Python). I am currently using this line of code in the HTML to obtain the CSV file:
<input type="file" accept="text/csv" id="mycsv">
Where in the Django project should I obtain the information from the CSV file, and how would I go about doing this? (I know the question is broad and doesn't give context for my specific project, but I figure that once I know how to access the data in the CSV I can figure the rest out).
Step 1: upload the file
# forms.py
from django import forms
class UploadFileForm(forms.Form):
file = forms.FileField()
Step 2: parse data and update database
# views.py
import csv
from .models import YourModel
def myview(request):
if request.method == "POST":
form = UploadFileForm(request.POST, request.FILES)
if form.is_valid():
reader = csv.reader(form.cleaned_data['file'])
for row in reader:
try:
some_instance = YourModel.objects.get_or_create(row[...])
...
Files uploaded by the user will go to the media folder that you have defined in your settings.py file.
You should be able to access user uploaded files in the media directory from your python code with something like this:
file_ = open(os.path.join(settings.MEDIA_ROOT, 'name_of_file'))
More info on MEDIA_ROOT and MEDIA_URL can be found here.
Related
I am trying to load the csv files I upload to my sqlite3 db. I am able to upload the csv into a folder but it doesn't appear on my db. The db table structure contains as many columns as the csv with the corresponding data types; but the column names in the db table structure are different as in the csv, as some of the columns names in the csv contain spaces and special characters (like accents for example).
My goal is to allow the user to upload a csv file that will be loaded to the db. I was also wondering if its possible to increment the db each time a new csv is loaded (for example: I upload a file now which is loaded to the db then the second time I add, the newer details are added to the db without deleting the previous ones). Apologies if I am unclear and asking too many things at once.
Here is my model, form, function (handle upload) & views .py files:
# models.py
class UploadForm(models.Model):
file = models.FileField()
class Meta:
db_table = "db_table"
# forms.py
class UploadForm(forms.ModelForm):
class Meta:
model = UploadForm
fields = "__all__"
# functions.py
def handle_uploaded_file(f):
with open('vps_app/static/upload/'+f.name, 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
# views.py
def index(request):
if request.method == 'POST':
Uploaded = UploadForm(request.POST, request.FILES)
if Uploaded.is_valid():
handle_uploaded_file(request.FILES['file'])
return HttpResponse("File uploaded successfuly")
else:
Uploaded = UploadForm()
return render(request, "index.html", {'form': Uploaded})
Thanks in advance.
You can't do this automatically, you need to handle it yourself, here is a sample for handle_uploaded_file
def handle_uploaded_file(f):
for row in csv.DictReader(f):
m = Model.objects.create(**row)
In my Django "sources" project I have the "bulletins" app.
I have a view "model_form_upload" inside of which I get the uploaded file name ("csvFilename") when I upload it.
views.py
from bulletins.forms import ErnageForm
from bulletins.models import ErnageModel
from bulletins.pretreatment import pretreatment
def model_form_upload(request):
if request.method == 'POST':
form = ErnageForm(request.POST, request.FILES)
if form.is_valid():
form.save()
for filename, file in request.FILES.items():
csvFilename = file.name
resultPretreatment = pretreatment()
print(resultPretreatment)
return redirect('proceed_FT')
else:
return redirect('upload_fail')
else:
form = ErnageForm()
return render(request, 'bulletins/upload_csv.html', {'form': form})
Beside that, I have a python script "pretreatment.py" that effectuates a bunch of pandas-dataframe transformations on my csv file.
pretreatment.py
def pretreatment(csvFileToTreat="..."
#... all the transformations with pandas dataframes...
df.to_csv(txtExportFilePath, index=False, header=None, sep='\t')
models.py
class ErnageModel(models.Model):
csv = models.FileField(upload_to="METEO ERNAGE/{0}/{1}".format(yearToTreat,monthToTreat))
My csvFilename will always be a string of this type : "Ernageyyyymm.csv".
My question here is : How can I collect my csvFilename variable out of the views.model_form_upload function to use its date information it in the pretreatment python script variables "csvFileToTreat" and "txtExportFilePath".
Those two ones are of the type "path/Ernage{0}{1}.format(yearToTreat,monthToTreat).
As I have to use it each month I would like to have my files uploaded in media/METEO ERNAGE/{0}/{1}".format(yearNumber,monthNumber) as you can see in the FileField upload_to and then everything happening in this directory. So the pretreatment charges the uploaded file and then export in it also.
I hope all this is as clear as possible, thank you for your help.
store csvFilename name in a session like this:
request.session['file']=csvFilename
and you can use this session where ever you want like this.
csvFilename =request.session['file']
I have a field where users upload an xlsx file In Django, however, im having trouble getting Django to return a simple text-based webpage from users uploading that xlsx file. Basically, I need Django to generate text from the cells of the xlsx file they uploaded. Im trying to use openpyxl to do this. How do I get Django to generate a new page with the results of the xlsx file after my script runs over it? Do I need to create a new template that redirects to it once the user submits the uploaded xlsx file? Here's an example.
# views.py
from django.shortcuts import render
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from .forms import UploadForm
import openpyxl
def index2(request):
if request.method != 'POST':
form = UploadForm()
else:
form = UploadForm(request.POST or None, request.FILES or None)
if form.is_valid():
form.save()
file = request.FILES['fileobj']
wb = openpyxl.load_workbook(file)
sheet = wb.get_sheet_by_name('Sheet1')
# script that runs overs the sheet here...
return HttpResponseRedirect(reverse('webpage with results'))
context = {'form': form}
return render(request, 'upload/upload_page.html', context)
I suggest decompose the problem. Right now you're trying to solve several things at once:
Upload the file
Save the file
Parse the file
Render a view of the file
I'd put 1) and 2) to the side and work on the XSLX loading/viewing part independently. For example, you could hard-code the path of the file in the view and get it rock solid that way. It's just development after all. Once it's good, remove the hard-coded file and connect up the real upload/save logic.
Realistically, you'll want to create a model class that represents say UploadedExcelFile. When the file is uploaded, create an instance and save it (i.e. a row in the database). Then you write a view that loads that model instance, which in turn can load the Excel file.
You might also want to consider ingesting the file rather than just storing on disk, or using some kind of async approach (e.g. django-q). This depends on a few factors, like what kind of information needs to be extracted from the file, is it editable etc. If it were me, and not knowing the bounds of your system, I'd do something like this:
Upload file
Store file in a temp area
Create a model object (i.e. database row) to represent the file and say the state is "processing".
Fire up a django-q job to ingest the file, giving it the filename
Return a view to this model.
The async job loads the file, parses the relevant aspects to json, stores the data in the model object (i.e. in the database), marks the model state as "ready".
The user can reload the page to see the data (or use websockets etc to tell it to reload).
I am trying to upload an image with django and so far i can't see the upload_to directory get created or any file source in the database field.
My Model.
class Person(models.Model):
photo = models.ImageField(upload_to='profiles/', null=True, blank=True)
My View
def create_profile(request):
if request.method == "POST":
form = PartialPersonForm(request.POST, request.FILES,)
addressForm = PartialAddressForm(request.POST)
When i upload and save, i dont get any sort of errors. Anyway i can debug this and find out if the photo field is set.
<div class="photo-upload">{{ form.photo|attr:"onchange:readURL(this);" }}
Unless you're doing something special with regard to media storage, you should set up MEDIA_ROOT to the directory where you want the uploaded files to be saved.
When dealing with file-type fields, you will need to following instructions in the Django documentation for FileField.storage.
In your view, you will need to save() the uploaded form data.
def create_profile(request):
if request.method == "POST":
form = PartialPersonForm(request.POST, request.FILES,)
addressForm = PartialAddressForm(request.POST)
form.save()
addressForm.save()
i believe you should create the upload_to directory yourself. Also make sure to set the appropriate permissions for the directory, otherwise it won't be able to write in it.
I need the Django Admin interface to accept administrator uploads of Excel files where the data in each Excel file is inserted into my database models. How can I make such an “Upload” button appear on a Django model admin page, where clicking the button asks the administrator to choose an .xls file, whose data then gets added to database once its upload is complete?
I have done this, but I just set up a simple view with a file upload (actually this makes more sense than adding it directly into a Django admin page, as one edit page = one model instance,and I assume that your excel contains multiple models).
in forms.py, a simple form with a file upload field
class ImportExcelForm(forms.Form):
file = forms.FileField(label= "Choose excel to upload")
in views.py, a view to process the upload
def test_flowcell(request):
c = RequestContext(request, {'other_context':'details here'})
if request.method == 'POST': # If the form has been submitted...
form = ImportExcelForm(request.POST, request.FILES) # A form bound to the POST data
if form.is_valid(): # All validation rules pass
excel_parser= ExcelParser()
success, log = excel_parser.read_excel(request.FILES['file'] )
if success:
return redirect(reverse('admin:index') + "pages/flowcell_good/") ## redirects to aliquot page ordered by the most recent
else:
errors = '* Problem with flowcell * <br><br>log details below:<br>' + "<br>".join(log)
c['errors'] = mark_safe(errors)
else:
c['errors'] = form.errors
else:
form = ImportExcelForm() # An unbound form
c['form'] = form
return render_to_response('sequencing/file_upload.html')
and as suggested in the other post use xlrd to read the data in from the excel file. I have a separate file ExcelParser.py for this
import xlrd
class ExcelParser(object, excel_name):
#transaction.commit_on_success
def read_excel(self):
wb = xlrd.open_workbook(excel_name)
...
do your parsing in here.....
...
(May I add, that excel is a terrible, and error prone way to import data. I do a lot of it at my work, and am trying to convince management that there are far better solutions.)
I'm not sure about the Django side of things, but you can use xlrd to read and manipulate Excel files. There is a free PDF which explains this called Working with Excel files in Python
django-import-export could be helpful.
It creates two buttons "import" and "export" for admin objects and permits select many types of extensions, including xls. It also show the data do be imported and asks to be confirmed before execute the execution.
You just need to include it in INSTALLED_APPS and create an import-export resource of the class you want to upload and a subclass of ImportExportModelAdmin related to the resource class created before to show buttons in admin.
more info at:
http://django-import-export.readthedocs.org/en/latest/getting_started.html
https://github.com/bmihelac/django-import-export.