I am trying to upload a CSV file into my Django model. Although the upload of the data works fine (all the rows get copied into the database), at the end Django returns a ValidationError ["'' value must be a decimal number."] error message.
From the local vars section of the error message I kind of get the reason - when the iteration reaches the end of the rows containing data, there is obviously no decimal number. So Django throws an error. However, I do not understand why, as there is always a last row after which there is no more data. I fiddled around a bit to try to find the problem. A method that worked is so copy the whole data from the original CSV into a new CSV - there was no error message any more. I would love to accomplish the whole process with the original CSV file and no error message! Would appreciate any help.
My CSV files are CSV UTF-8 and they are saved in Excel
models.py
from django.db import models
class Testdata3(models.Model):
key = models.CharField(max_length=100, primary_key=True)
assetclass = models.CharField(max_length=25)
value = models.DecimalField(max_digits=25,decimal_places=10)
performance = models.DecimalField(max_digits=25,decimal_places=10)
def __str__(self):
return self.key
views.py
from django.shortcuts import render
from .models import Testdata3
import csv, io
from django.contrib import messages
def file_upload(request):
template = "upload.html"
prompt = {
'order': 'Order of the CSV should be "placeholder_1", "placeholder_2", "placeholder_3" '
}
if request.method == "GET":
return render(request, template, prompt)
csv_file = request.FILES['file']
if not csv_file.name.endswith('.csv'):
messages.error(request, 'This is not a csv file')
data_set = csv_file.read().decode('UTF-8')
io_string = io.StringIO(data_set)
next(io_string)
for column in csv.reader(io_string, delimiter=';', quotechar="|"):
_, created = Testdata3.objects.update_or_create(
key = column[0],
defaults = {
'key' : column[0],
'assetclass' : column[10],
'value' : column[16],
'performance' : column[18],
}
)
context = {}
return render(request, template, context)
upload.html
{% if messages %}
{% for message in messages %}
<div>
<strong>{{message|safe}}</strong>
</div>
{% endfor %}
{%else %}
{{ order }}
<form method="post" enctype="multipart/form-data">
{% csrf_token %}
<label>Upload a file</label>
<input type="file" name="file">
<p>Only accepts CSV files</p>
<button type="submit">Upload</button>
</form>
{% endif %}
I've found a solution! I included an if-statement, which checks if the current iteration has only empty values, and if yes it jumps to the next iteration
# ...
views.py
for column in csv.reader(io_string, delimiter=';', quotechar="|"):
# Check for empty rows in CVS and jump to next iteration if empty
if all(elem == "" for elem in column):
next
else:
_, created = Testdata3.objects.update_or_create(
key = column[0],
defaults = {
'key' : column[0],
'assetclass' : column[10],
'value' : column[16],
'performance' : column[18],
}
)
# ...
Related
Right now I have my urls.py set up like this:
urlpatterns = [
...
path('dividends/<str:month>/', views.DividendView.as_view(), name='dividendview'),
path('dividends/', views.DividendView.as_view(), name='dividendview'),
]
What I would like is to have the 'month' parameter optional and default to today's month. Right now I have my views.py set up as
class DividendView(ListView):
model = Transaction
template_name = 'por/dividends.html'
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
divs = Dividends()
month = self.kwargs['month']
context['month'] = get_month(month)
return context
def get_month(month):
if month:
return month
else:
return datetime.today().month
and my dividends.html file as
{% extends 'base.html' %}
{% load static %}
{% block title %}Dividends{% endblock %}
{% block content %}
{{ month }}
{% endblock %}
If I navigate to /dividends/Oct/ (or any other month) it works fine, but if I just go to /dividends/ it gives me
KeyError: 'month'
What am I doing wrong and how might I go about fixing it?
First, you need to check if the kwarg 'month' exists and then assign the month value else it will raise keyError.
Views.py
class DividendView(ListView):
model = Transaction
template_name = 'por/dividends.html'
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
divs = Dividends()
if 'month' in self.kwargs: # check if the kwarg exists
month = self.kwargs['month']
else:
month = datetime.today().month
context['month'] = month
return context
You can do it in very simply way and you dont need to define two endpoints in you urls.py
(?P<month>\w+|)
So your url will be :-
path('dividends/(?P<month>\w+|)/', views.DividendView.as_view(), name='dividendview'),
I built an app which allows the user to upload a file.
On the other hand,
I have some python script which takes a text file, convert it to CSV and do pandas. This script works perfectly when running in the terminal.
Now I want to apply that python script to the file upload in Django and show that file in httpResponse and make available to download it.
python script
import csv
import pandas as pd
df = pd.read_csv('raw_df.txt', delimiter = '\t' , usecols=['Sample','Cohort','Metabolite Name','Intensity'])
df = df[df['Cohort'].str.contains("std")]
df = df.groupby(['Cohort', 'Metabolite Name'])['Intensity'].sum().reset_index()
df = df[['Cohort','Intensity']]
c = 'Cohort'
s = df.set_index([c, df.groupby(c).cumcount() + 2]).Intensity
df = s.unstack().add_prefix('Intensity').reset_index()
df.to_csv()
print df;
views.py
from django.shortcuts import render, redirect
from django.conf import settings
from django.core.files.storage import FileSystemStorage
from uploads.core.models import Document
from uploads.core.forms import DocumentForm
def home(request):
documents = Document.objects.all()
return render(request, 'core/home.html', { 'documents': documents })
def model_form_upload(request):
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
form.save()
return redirect('home')
else:
form = DocumentForm()
return render(request, 'core/model_form_upload.html', {
'form': form
})
Models.py
class Document(models.Model):
document = models.FileField(upload_to='documents/')
template-upload page
{% extends 'base.html' %}
{% block content %}
<form method="post" enctype="multipart/form-data">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Upload</button>
</form>
<p>Return to home</p>
{% endblock %}
Wrap your file processing in a method:
import csv
import pandas as pd
def process_file(file_handle):
df = pd.read_csv(file_handle, delimiter = '\t' , usecols['Sample','Cohort','Metabolite Name','Intensity'])
df = df[df['Cohort'].str.contains("std")]
df = df.groupby(['Cohort', 'Metabolite Name'])['Intensity'].sum().reset_index()
df = df[['Cohort','Intensity']]
c = 'Cohort'
s = df.set_index([c, df.groupby(c).cumcount() + 2]).Intensity
df = s.unstack().add_prefix('Intensity').reset_index()
return df.to_csv()
in your view:
...
if form.is_valid():
document = form.save()
# call to the new method
csv = process_file(document.document)
response = HttpResponse(csv, content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename=result.csv'
return response
...
I want to create a list in HTML of locations from JSON API url in python.
#app.route('/api')
def api():
url = urlopen('https://api.openaq.org/v1/locations?country=GB').read()
#encoded we do not need it
#encoded_data = json.dumps(url)
#create variables
array = []
data = {}
#decoded
decoded_data = json.loads(url)
#we search for result each entry
for i in decoded_data["results"]:
#append every entry to an array
array.append(i["location"])
#we create a dictionary from that array created which has a variable (for jinja2) called location
data = [dict(location=array)]
return render_template('api.html', data=data)
But instead of receiving each element, I get this:
[u'Aberdeen', u'Aberdeen Union Street Roadside', u'Aberdeen Wellington Road', u'Armagh Roadside', u'Aston Hill', u'Auchencorth Moss', u'Ballymena Ballykeel', u'Barnsley Gawber', u'Barnstaple A39', u'Bath Roadside', u'Belfast Centre', u"Belfast Stockman's Lane", u'Billingham', u'Birkenhead Borough Road', u'Birmingham A4540 Roads...
Edit: Template
{% if data %}
<ul>
{% for d in data %}
<li>{{ d.location }}</li>
{% endfor %}
</ul>
{% else %}
<p class="lead">
You should not see this msg, otherwise, check the code again.
</p>
{% endif %}
I broke my answer down a bit because I didn't want to activate flask.
import requests
def api():
res = requests.get('https://api.openaq.org/v1/locations?country=GB')
data = res.json()['results']
return data
#app.route('/api')
def api():
res = requests.get('https://api.openaq.org/v1/locations?country=GB')
try:
data = res.json()['results']
except KeyError:
data = None
# this is the logic that you use in your template, I moved it out here
# cause i don't want to start a flask instance
for d in data:
print d['location']
return render_template('api.html', data=data)
api()
Basically I use the requests module which can return a json. I pass the results to the data varible. I used a for loop to demo how it would work in your template. Basically pass in the data as a dictionary and get the location via iteration d['location']
So the code to use is
import requests
#app.route('/api')
def api():
res = requests.get('https://api.openaq.org/v1/locations?country=GB')
try:
data = res.json()['results']
except KeyError:
data = None
return render_template('api.html', data=data)
You are converting the array to a dict, but then you are putting the dict inside an array of length 1, with the only object being the dict. The issue is, your template is then expecting each element in the array to be a dictionary, with a "location" field.
You either can remove the square brackets from the conversion data = dict(location=array) and then update your template to just do for d in data.location, or you can update your append call to append a dictionary item instead of a string: array.append({"location": i["location"]})
Couple of things:
url is a bytes object, which will not work with json.loads(str). So you'll have to convert it to a string either by doing json.loads(str(url,'utf-8')) or the method suggested by #Mattia
#louhoula is correct. But, in case you are expecting data to be a list of dictionaries each containing a location key (that's the idea I get by looking at your template), then you should change d.location in your template to :
{% if 'location' in d.keys(): %}
{{ d['location'] }}
{% else %}
<p class="lead">
You should not see this msg, otherwise, check the code again.
</p>
{% endif %}
Try this:
import urllib.request, json
url = 'https://api.openaq.org/v1/locations?country=GB'
response = urllib.request.urlopen(url);
decoded_data = json.loads(response.read().decode("utf-8"))
My first foray into Django, allows the user to input a day of the week, and searches my database for restaurants open on that given day. Right now, the restaurant objects (Resto) have a days_open attribute, with each day separated by a comma (Monday, Tuesday, etc...).
When I input a day, the resulting page only displays the title and '[]' and I can't seem to get it to return the list of Resto objects. What is displaying the square brackets, and how do I go about fixing it to display the results of the search?
The code is below- my apologies if I neglected to include any relevant bits.
my forms.py:
from django import forms
from .models import Resto
class RestoSearch(forms.ModelForm):
class Meta:
model = Resto
fields = ('title', 'description', 'opening_hour', 'closing_hour')
models.py:
from django.db import models
class Resto(models.Model):
title = models.CharField(max_length=300)
description = models.TextField()
opening_hour = models.TimeField(auto_now=False, auto_now_add=False, null=True)
closing_hour = models.TimeField(auto_now=False, auto_now_add=False, null=True)
days_open = models.TextField(blank=True)
views.py:
from django.shortcuts import render
from django.http import Http404
from django.shortcuts import HttpResponse
from belize.models import Resto
from django.core.exceptions import *
from .forms import RestoSearch
def index(request):
return render(request, 'form.html')
def search(request):
form = RestoSearch()
if request.method == 'POST':
search_id=request.POST.get('textfield', None)
try:
#I think this is where my problem is
available = Resto.objects.filter(days_open = search_id)
html = ("<H1>Hello</H1>", available)
return HttpResponse(html)
except Resto.DoesNotExist:
return HttpResponse("Please try another day")
else:
return render(request, 'belize/form.html')
def restaurant_detail(request, id):
try:
restaurant = Resto.objects.get(id=id)
except Resto.DoesNotExist:
raise Http404('This restaurant does not exist')
return render(request, 'belize/restaurant_detail.html', {
'restaurant': restaurant,
})
template form.html:
<form method="POST" action="/search/">
{% csrf_token %}
<input type="text" name="textfield">
<button type="submit">Enter a day of the week</button>
</form>
I presume what you are trying to show is the RestoForm in that case the index method is not correct. It should be
def index(request):
form = RestoForm()
return render(request, 'form.html', {'form': form })
And then your template should change as
<form method="POST" action="/search/">
{% csrf_token %}
{{ form }}
<button type="submit">Enter a day of the week</button>
</form>
For additional details please see the examples at https://docs.djangoproject.com/en/1.9/topics/forms/#the-template
The [] means that your .filter() returned no results, its not surprising as you have a few issues with your code, lets start from the top:
You are declaring a form that you never use.
You are trying to catch an exception that is never raised by .filter()
You filter condition will only work for exact matches.
I've annotated your code as well:
def search(request):
form = RestoSearch() # You aren't using this form anywhere in your code?
if request.method == 'POST':
# Where is 'textfield' coming from?
search_id = request.POST.get('textfield', None)
try:
# If search id is "Tuesday", and a restaurant is
# open on monday and tuesday, so it has "Monday,Tuesday"
# in the days_open field, then this search will not
# return any results, because its looking for an exact
# match
available = Resto.objects.filter(days_open=search_id)
html = ('<H1>Hello World</H1>', available)
return HttpResponse(html)
except Resto.DoesNotExist:
# This exception is not raised by .filter(),
# .filter() will return an empty list, [] if no results are found
# so this entire try/except is not doing anything
return HttpResponse("Please try another day")
else: # in your code, this else is not indented correctly
return render(request, 'belize/form.html')
So there is a lot going on here, lets try something simple, starting with the template:
{% if results %}
{% for restaurant in results %}
{{ restaurant }}
{% endfor %}
{% else %}
Sorry, no results for your search. Try again.
{% endif %}
<form>
{{ form }}
<input type="submit" name="Search" />
</form>
Next, the search form:
class SearchForm(forms.Form):
search_field = forms.CharField('Search', strip=True)
Finally the view:
from django.db.models import Q
def search(request):
form = SearchForm(request.GET)
results = [] # Set results to an empty list
if form.is_valid():
needle = form.cleaned_data['search_field'].capitalize()
results = Resto.objects.filter(Q(days_open__startswith='{},'.format(needle)) |
Q(days_open__endswith=',{}'.format(needle)) |
Q(days_open__contains=',{},'.format(needle)) |
Q(days_open='{},'.format(needle)) |
Q(days_open='{}'.format(needle)))
return render(request, 'search.html', {'results': results, 'form': form})
Lets assume the user entered 'Thursday' as a search field. In this view you are searching for all restaurants whose days_open field:
Either starts with Thursday, or
Ends with ,Thursday or
Contains ,Thursday, in the middle or
Has the value Thursday, or
Has the value Thursday
Your template will then only show the results if there are any values to display; since an empty list [] is false, then the {% if results %} condition will fail, so on empty lists the template will display the error notice instead.
In your view, you only do the database check if someone enters something in the search field, that's what if form.is_valid(): does. In django, by default all form fields are required - so a blank form will return an error. Using this trick, we make sure we only search if someone enters a value in the search box.
The main action happens with all the Q() calls. A Q object is a way to do multiple queries and chain them together. It is used whenever you want to do an "or" or "and" type query. Our search is an "or" type query, because we want to return any results if the value in days_open matches any number of conditions.
I'm trying to understand how to edit or update a model. I have tried several scenarios which sometimes give an error message: 405 Method Not Allowed - The method POST is not allowed for this resource. Below is my code:
The Python Models:
import os
import webapp2
import wsgiref.handlers
from google.appengine.ext import db
from google.appengine.ext.webapp import template
class MessageModel(db.Model):
content = db.StringProperty(multiline=True)
date = db.DateTimeProperty(auto_now_add=True)
class Message(webapp2.RequestHandler):
def get(self):
doRender(self,'message.htm')
def post(self):
m = MessageModel()
m.content = self.request.get('content')
m.put()
self.redirect('/view')
class View(webapp2.RequestHandler):
def get(self):
que = db.Query(MessageModel)
messageview_list = que.fetch(999)
doRender(self,
'view.htm',
{'messageview_list': messageview_list })
class Edit(webapp2.RequestHandler):
def get(self):
doRender(self,'edit.htm')
def post(self):
updated_content = self.request.get('content')
content_query = db.GqlQuery("SELECT * "
"FROM MessageModel "
"ORDER BY date DESC LIMIT 1")
messageview_list = content_query.fetch(1)
m = MessageModel()
m.content = self.request.get(updated_content)
m.put()
doRender(self,
'edit.htm',
{'messageview_list': messageview_list })
class Main(webapp2.RequestHandler):
def get(self):
doRender(self,'index.htm')
def doRender(handler, tname = 'index.htm', values = { }):
temp = os.path.join(
os.path.dirname(__file__),
'templates/' + tname)
if not os.path.isfile(temp):
return False
newval = dict(values)
newval['path'] = handler.request.path
outstr = template.render(temp, newval)
handler.response.out.write(outstr)
return True
app = webapp2.WSGIApplication([('/', Main),
('/message', Message),
('/view', View),
('/edit', Edit)],
debug=True)
The HTML Form:
{% for messageview in messageview_list %}
<form method="post" action="/edit">
<p>
<textarea name="message" rows="3" cols="60" MAXLENGTH=60>
{{ messageview.content }}</textarea>
<br>
<input type="submit" value="Update"/>
</p>
</form>
{% ifnotequal error None %}
<p>
{{ error }}
</p>
{% endifnotequal %}
{% endfor %}
I am assuming the indentation is due to copy/paste, but make sure that the post() and get() functions are actually indented inside of your class.
In your form, you have <textarea name="message" rows="3" cols="60" MAXLENGTH=60>, but in your def post() you use updated_content = self.request.get('content'), which is looking for the content keyword in the request. Also, your edit doesn't look like it is doing what you want it to do. In order to edit an entity, the basic outline of the process is 1.) Retrieve the entity (so do as you do, query using some parameter); 2.) Modify the properties of the entity however you want; and 3.) put() the entity back in the datastore.
From your code, it looks like you are retrieving the last entity entered into the datastore, but then creating a new model instead of editing that one (assuming that is what you want to do - not quite sure if that is accurate :) ). If you are looking to modify the entity that is returned, this should work:
def post(self):
updated_content = self.request.get('message')
content_query = db.GqlQuery("SELECT * "
"FROM MessageModel "
"ORDER BY date DESC LIMIT 1")
# Your query will always return just one entity (due to the LIMIT),
# but you can use get() here instead of fetch(1)
latest_model = content_query.get()
# Update the model's content property
latest_model.content = updated_content
latest_model.put()
# Assuming you want to output that model, you'd output it now
doRender(self,
'edit.htm',
{'messageview_list': latest_model })