How can i create a xml file in django where the file is to contain objects from a queryset?
def my_serialize(request):
from django.core import serializers
data = serializers.serialize('xml', Student.objects.filter(Q(name__startswith='A')),
fields=('name','dob'))
from django.core.files import File
f = open('content.xml', 'w')
myfile = File(f)
myfile.write(data)
myfile.close()
After i call the above function, my content file remains empty, there is no data that gets written in it.
from django.core import serializers
data = serializers.serialize("xml", SomeModel.objects.all())
Take a look at the Serialization documentation.
I don't use version 1.3, so I'm not sure how it works, but could it be that the file is not actually being opened? Could adding myfile.open('w') work? Of course, this may be handled in the File class's init function. Anyway, try giving it a shot and commenting your results.
Update:
BTW, I came up with the idea from looking at the docs. Maybe they'll be able to help too.
http://docs.djangoproject.com/en/1.3/ref/files/file/
Related
I am new in work with JSON, so sorry in advance for the stupid question.
I want to write JSON with the variable in the value field. It looks like this:
def print_json(user_name):
opened_json = open('way/to/json/file')
tmp = json.load(opened_json)
res = tmp(['path_to_folder'](user_name))
print(res)
def main(user_name):
print_json(user_name)
main('user')
It is JSON:
{"path_to_folder": "/Users/" + user_name + "/my_folder/"}
Awaiting for that output:
/Users/user/my_folder/
Please, tell me if any solution here exists.
Thanks in advance!
EDIT: My problem, that I can't add variable to JSON correctly. It marked red. Wrong syntax, when I try to concat.
What you want isn't directly possible in JSON, because it doesn't support "templating".
One solution would be to use a templating language such as Jinja to write a JSON template, then load this file without the json library and fill in the values using Jinja, and finally use json.loads to load a dictionary from your rendered string.
Your json-like file could look something like this:
{"path_to_folder": "/Users/{{ user_name }}/my_folder/"}
Your Python code:
import json
from jinja2 import Environment, FileSystemLoader
env = Environment(
FileSystemLoader("path/to/template")
)
template = env.get_template("template_filename.json")
def print_json(username):
return json.loads(
template.render(user_name=username)
)
...
In fact, if this is a simple one-time thing, it might even be better to use Python's built-in templating. I would recommend old-style formatting in the case of JSON, because otherwise you'll have to escape a lot of braces:
JSON file:
{"path_to_folder": "/Users/%(user_name)s/my_folder/"}
"Rendering":
with open("path/to/json") as f:
rendered = json.loads(f.read() % {"user_name": username})
In order to become more familiar with django, I decided to build a website which is gonna let a user upload a csv file, which is then gonna be converted to excel and the user will be able to download it.
In order to achieve that I created a modelform with one model FileField called csv_file as shown below:
#models.py
class CSVUpload(models.Model):
csv_file = models.FileField(upload_to="csvupload/")
def __str__(self):
return self.csv_file
#forms.py
class CsvForm(forms.ModelForm):
class Meta:
model = CSVUpload
fields = ('csv_file', )
and the corresponding view is:
from django.shortcuts import render, redirect
import pandas as pd
import os
#converts file from csv to excel
def convertcsv2excel(filename):
fpath = os.path.join(settings.MEDIA_ROOT + "\csvupload", filename)
df = pd.read_csv(fpath)
newfilename = str(filename) +".xlsx"
newpathfile = os.path.join(settings.MEDIA_ROOT, newfilename)
df.to_excel(newpathfile, encoding='utf-8', index=False)
return newfilename
def csvtoexcel(request):
if request.method == 'POST':
form = CsvForm(request.POST, request.FILES)
if form.is_valid():
form.save()
print(form.cleaned_data['csv_file'].name)
convertcsv2excel(form.cleaned_data['csv_file'].name)
return redirect('exceltocsv')
else:
form = CsvForm()
return render(request, 'xmlconverter/csvtoexcel.html',{'form': form})
right now as you can see I am using Pandas in order to convert the csv file to excel inside the views.py file. My question is, is there a better way to do it (for instance in the form or model module) in order to make the excel file more effectively downloadable?
I appreciate any help you can provide!
First, I want to point out that your example demonstrates an arbitrary file upload vulnerability. Pandas does not validate the format of the file for you, so as an attacker, I can simply upload something like malware.php.csv to your conversion script, and any malicious code I include will remain intact. Since you aren't validating that this file's contents are, in fact, in CSV format, then you are giving users a means to directly upload a file with an arbitrary extension and possibly execute code on your website. Since you are rendering the xlsx format on the webpage the way you are, there's a good chance someone could abuse this. If this is just your own personal experiment to help yourself get familiar, that's one thing, but I strongly recommend against deploying this in production. What you are doing here is very dangerous.
As for your more immediate problem, I'm not personally familiar with Django, but this looks very similar to this question: Having Django serve downloadable files
In your case, you do not want to actually save the file's contents to your server but rather you want to process the file contents and return it in the body of the response. The django smartfile module looks to be exactly what you want: https://github.com/smartfile/django-transfer
This provides components for Apache, Nginx, and lighttpd and should allow you to provide a means to provide files in the response immediately following a request to upload/convert the file. I should emphasize that you need to be very careful about where you save these files, validating their contents, ensure end-users cannot browse to or execute these files under the web server context, and that they are deleted immediately after the response and file is successfully sent.
Someone more familiar with Django can feel free to correct me or provide a usable code example, but this kind of functionality, in my experience, is how you introduce code execution into your site. It's usually a bad idea.
I'm trying to modify a csv that is uploaded into my flask application. I have the logic that works just fine when I don't upload it through flask.
import pandas as pd
import StringIO
with open('example.csv') as f:
data = f.read()
data = data.replace(',"', ",'")
data = data.replace('",', "',")
df = pd.read_csv(StringIO.StringIO(data), header=None, sep=',', quotechar="'")
print df.head(10)
I upload it to flask and access it using
f = request.files['data_file']
When I run it through the code above, replacing open('example.csv') with open(f), I get the following error
coercing to Unicode: need string or buffer, FileStorage found
I have figured out that the problem is the file type here. I can't use open on my file because open is looking for a file name and when the file is uploaded to flask it is the instance of the file that is being passed to the open command. However, I don't know how to make this work. I've tried skipping the open command and just using data = f.read() but that doesn't work. Any suggestions?
Thanks
FileStorage is a file-like wrapper around the incoming data. You can pass it directly to read_csv.
pd.read_csv(request.files['data_file'])
You most likely should not be performing those replace calls on the data, as the CSV module should handle that and the naive replacement can corrupt data in quoted columns. However, if you still need to, you can read the data out just like you were before.
data = request.files['data_file'].read()
If your data has a mix of quoting styles, you should fix the source of your data.
Answering my own question in case someone else needs this.
FileStorage objects have a .stream attribute which will be an io.BytesIO
f = request.files['data_file']
df = pandas.read_csv(f.stream)
I have a file bound to a form in this manner:
(forms.py)
class UploadFileForm(forms.Form):
wbfile = forms.FileField(label='Upload workbook' , help_text='Please Ensure file is in .xlsx format')
Now I have can access this in a view function using request.FILES['wbfile']. But I want to send this file to a template and then to another view function. So I bound it to a form like this:
f = form.fields['wbfile']
Now I want to save this file in the disk, but how do I access this file, this is what I am trying:
f = form.fields['file'].value()
with open(/tmp/xyz) as destination:
contents = f.read()
destination.write(contents)
But this throws an error saying: 'FileField' object has no attribute 'value'.
This is what for.fields[wbfile] shows:
<django.forms.fields.FileField object at 0x7f91ff1c49d0>
Hence the file is definitely bound to the form.
Please help and forgive me if the doubt is too obvious, I am a beginner!
I think there is everythink you need in the Django documentation.
For FileField : https://docs.djangoproject.com/en/dev/ref/models/fields/#filefield
Try using YourFileField.storage and YourFileField.path to have access to the file.
Then with this doc : https://docs.djangoproject.com/en/dev/ref/files/storage/
You can use open to open your file in memory. I guess it can give a code like this :
storage, path = YourFileField.storage, YourFileField.path
File f = storage.open(path)
I don't really understand what you want to do next, but if you want to save the file somewhere else, I guess you can probably use :
f.save(path) #or something similar, haven't tested any code like this
I would like to store large dataset generated in Python in a Django model. My idea was to pickle the data to a string and upload it to FileField of my model. My django model is:
#models.py
from django.db import models
class Data(models.Model):
label = models.CharField(max_length=30)
file = models.FileField(upload_to="data")
In my Python program I would like to do the following:
import random, pickle
data_entry = Data(label="somedata")
somedata = [random.random() for i in range(10000)]
# Next line does NOT work
#data_entry.file.save(filename, pickle.dumps(somedata))
How should I modify the last line to store somedata in file preserving the paths defined with upload_to parameter?
Based on the answers to the questions I came up with the following solution:
from django.core.files.base import ContentFile
import pickle
content = pickle.dumps(somedata)
fid = ContentFile(content)
data_entry.file.save(filename, fid)
fid.close()
All of it is done on the server side and and users are NOT allowed to upload pickles. I tested it and it works all fine, but I am open to any suggestions.
In you database the file attribute is just a path to the file. So, since you are not doing an actual upload you need to store the file on the disk and then save the path in database.
f = open(filename, 'w')
pickle.dump(somedata, f)
f.close()
data_entry.file=filename
data_entry.save()
Might you not be better off storing your data in a text field? It's not a file upload, after all.
I've never done this, but based on reading a bit of the relevant code, I'd start by looking into creating an instance of django.core.files.base.ContentFile and assigning that as the value of the field.
NOTE: See other answers and comments below - old info and broken links removed (can't delete a once-accepted answer).
Marty Alchin has a section on this in chapter 3 of Pro Django, review here.