Example:
A system that processes different types of csv files. The different types of csv files have different content inside.
So the current way im thinking about modeling this is:
File model that stores some generic info about each file processed, like the name, date etc.
class File(models.Model):
name = models.CharField()
Then models that stores the content for each type of file and these models link back to the File model:
class Example1File(models.Model):
file = models.ForeignKey(File, related_name=example_file_1_content)
... (all columns for this file)
class Example2File(models.Model):
file = models.ForeignKey(File, related_name=example_file_2_content)
... (all columns for this file)
Is there an easy way for example to loop through the files table and get the content for each file and django would just know in which table to go look?
for file in File.objects.all():
file.???
Or would I only be able to do this by storing a type field on the file and then getting the content with the reverse relationship like this.
for file in File.objects.all():
if file.type == FileTypes.ExampleFile1:
file.example_file_1_content
But what if there are a lot of different file types? Im still in the process of learning about how django handles relationships/reverse relationships so I might just not be there yet or the way I am modeling this is not optimal.
Thanks in advance.
Related
In order to become more familiar with django, I decided to build a website which is gonna let a user upload a csv file, which is then gonna be converted to excel and the user will be able to download it.
In order to achieve that I created a modelform with one model FileField called csv_file as shown below:
#models.py
class CSVUpload(models.Model):
csv_file = models.FileField(upload_to="csvupload/")
def __str__(self):
return self.csv_file
#forms.py
class CsvForm(forms.ModelForm):
class Meta:
model = CSVUpload
fields = ('csv_file', )
and the corresponding view is:
from django.shortcuts import render, redirect
import pandas as pd
import os
#converts file from csv to excel
def convertcsv2excel(filename):
fpath = os.path.join(settings.MEDIA_ROOT + "\csvupload", filename)
df = pd.read_csv(fpath)
newfilename = str(filename) +".xlsx"
newpathfile = os.path.join(settings.MEDIA_ROOT, newfilename)
df.to_excel(newpathfile, encoding='utf-8', index=False)
return newfilename
def csvtoexcel(request):
if request.method == 'POST':
form = CsvForm(request.POST, request.FILES)
if form.is_valid():
form.save()
print(form.cleaned_data['csv_file'].name)
convertcsv2excel(form.cleaned_data['csv_file'].name)
return redirect('exceltocsv')
else:
form = CsvForm()
return render(request, 'xmlconverter/csvtoexcel.html',{'form': form})
right now as you can see I am using Pandas in order to convert the csv file to excel inside the views.py file. My question is, is there a better way to do it (for instance in the form or model module) in order to make the excel file more effectively downloadable?
I appreciate any help you can provide!
First, I want to point out that your example demonstrates an arbitrary file upload vulnerability. Pandas does not validate the format of the file for you, so as an attacker, I can simply upload something like malware.php.csv to your conversion script, and any malicious code I include will remain intact. Since you aren't validating that this file's contents are, in fact, in CSV format, then you are giving users a means to directly upload a file with an arbitrary extension and possibly execute code on your website. Since you are rendering the xlsx format on the webpage the way you are, there's a good chance someone could abuse this. If this is just your own personal experiment to help yourself get familiar, that's one thing, but I strongly recommend against deploying this in production. What you are doing here is very dangerous.
As for your more immediate problem, I'm not personally familiar with Django, but this looks very similar to this question: Having Django serve downloadable files
In your case, you do not want to actually save the file's contents to your server but rather you want to process the file contents and return it in the body of the response. The django smartfile module looks to be exactly what you want: https://github.com/smartfile/django-transfer
This provides components for Apache, Nginx, and lighttpd and should allow you to provide a means to provide files in the response immediately following a request to upload/convert the file. I should emphasize that you need to be very careful about where you save these files, validating their contents, ensure end-users cannot browse to or execute these files under the web server context, and that they are deleted immediately after the response and file is successfully sent.
Someone more familiar with Django can feel free to correct me or provide a usable code example, but this kind of functionality, in my experience, is how you introduce code execution into your site. It's usually a bad idea.
This is my model.
class Product(models.Model)
id = models.AutoField(max_length=10, primary_key=True)
name = models.CharField(max_length=60)
summary = models.TextField(max_length=200)
image = models.FileField(upload_to='product_images', blank=True)
I know how to import data from csv in django model. But I also have image field here.
Can I upload files to models from the method of adding data to models from csv? How?
I am using this method for importing them: http://mitchfournier.com/2011/10/11/how-to-import-a-csv-or-tsv-file-into-a-django-model/
I have images stored in a folder in my system.
FileField is just a reference to a file. It does not put a file into the database as is sometimes mistakenly believed.
Assuming that you have already got the code written for reading through the CSV and fetching the location of the file. Then all you need to do is to follow the example given in FileField.save()
This method takes a filename and file contents and passes them to the
storage class for the field, then associates the stored file with the
model field. If you want to manually associate file data with
FileField instances on your model, the save() method is used to
persist that file data.
Takes two required arguments: name which is the name of the file, and
content which is an object containing the file’s contents.
Something like this:
p = Product()
p.image.save('name from csv',open('path from csv'))
I solved it by saving the name of the image in the csv and just uploading the name in the field like this:
...
product.image = "product_images" + namefromcsv
then I uploaded all the images in the media folder inside product_images .
And it worked very well.
I tried uploading the image with product.image.save() option with File class but it didn't work for me.
I'm trying to create a rails app that is a CMS for a client. The app currently has a documents class that uploads the document with paperclip.
Separate to this, we're running a python script that accesses the database and gets a bunch of information for a given event, creates a proposal word document, and uploads it to the database under the correct event.
This all works, but the app does not recognize the document. How do I make a python script that will correctly upload the document such that paperclip knows what's going on?
Here is my paperclip controller:
def new
#event = Event.find(params[:event_id])
#document = Document.new
end
def create
#event = Event.find(params[:event_id])
#document = #event.documents.new(document_params)
if #document.save
redirect_to event_path(#event)
end
end
private
def document_params
params.require(:document).permit(:event_id, :data, :title)
end
Model
validates :title, presence: true
has_attached_file :data
validates_attachment_content_type :data, :content_type => ["application/pdf", "application/msword"]
Here is the python code.
f = open(propStr, 'r')
binary = psycopg2.Binary(f.read())
self.cur.execute("INSERT INTO documents (event_id, title, data_file_name, data_content_type) VALUES (%d,'Proposal.doc',%s,'application/msword');" % (self.eventData[0], binary))
self.con.commit()
You should probably use Ruby to script this since it can load in any model information or other classes you need.
But assuming your requirements dictate the use of python, be aware that Paperclip does not store the documents in your database tables, only the files' metadata. The actual file is stored in your file system in the /public dir by default (could also be s3, etc depending on your configuration). I would make sure you were actually saving the file to the correct anticipated directory. The default path according to the docs is:
:rails_root/public/system/:class/:attachment/:id_partition/:style/:filename
so you will have to make another sql query to retrieve the id of your new record. I don't believe pdfs have a :style attribute since you don't use imagicmagick to resize them, so build a path that looks something like this:
/public/system/documents/data/000/000/123/my_file.pdf
and save it from your python script.
Right now, I have a Django application with an import feature which accepts a .zip file, reads out the csv files and formats them to JSON and then inserts them into the database. The JSON file with all the data is put into temp_dir and is called data.json.
Unfortunatly, the insertion is done like so:
Building.objects.all().delete()
call_command('loaddata', os.path.join(temp_dir, 'data.json'))
My problem is that all the data is deleted then re-added. I need to instead find a way to update and add data and not delete the data.
I've been looking at other Django commands but I can't seem to find out that would allow me to insert the data and update/add records. I'm hoping that there is a easy way to do this without modifying a whole lot.
If you loop through your data you could use get_or_create(), this will return the object if it exist and create it if it doesn't:
obj, created = Person.objects.get_or_create(first_name='John', last_name='Lennon', defaults={'birthday': date(1940, 10, 9)})
I would like to store large dataset generated in Python in a Django model. My idea was to pickle the data to a string and upload it to FileField of my model. My django model is:
#models.py
from django.db import models
class Data(models.Model):
label = models.CharField(max_length=30)
file = models.FileField(upload_to="data")
In my Python program I would like to do the following:
import random, pickle
data_entry = Data(label="somedata")
somedata = [random.random() for i in range(10000)]
# Next line does NOT work
#data_entry.file.save(filename, pickle.dumps(somedata))
How should I modify the last line to store somedata in file preserving the paths defined with upload_to parameter?
Based on the answers to the questions I came up with the following solution:
from django.core.files.base import ContentFile
import pickle
content = pickle.dumps(somedata)
fid = ContentFile(content)
data_entry.file.save(filename, fid)
fid.close()
All of it is done on the server side and and users are NOT allowed to upload pickles. I tested it and it works all fine, but I am open to any suggestions.
In you database the file attribute is just a path to the file. So, since you are not doing an actual upload you need to store the file on the disk and then save the path in database.
f = open(filename, 'w')
pickle.dump(somedata, f)
f.close()
data_entry.file=filename
data_entry.save()
Might you not be better off storing your data in a text field? It's not a file upload, after all.
I've never done this, but based on reading a bit of the relevant code, I'd start by looking into creating an instance of django.core.files.base.ContentFile and assigning that as the value of the field.
NOTE: See other answers and comments below - old info and broken links removed (can't delete a once-accepted answer).
Marty Alchin has a section on this in chapter 3 of Pro Django, review here.