Develop Reference

Apply filter to nested reverse foreign key relationship using queryset

Context
I am trying to filter a list of objects based on the value of a reverse foreign key attribute.
I was able to solve it at the view level but, but other attempts to solve using ORM feature result in additional queries.
The outcome I want to is queryset with all objects, but related fkey objects are filtered inside each object.
Sample Models
class Student(models.Model):
name = models.CharField(max_length=128)
class Subject(models.Model):
title = models.CharField(max_length=128)
class Grade(models.Model):
student = models.ForeignKey("Student", related_name="grades", on_delete=models.CASCADE)
subject = models.ForeignKey("Subject", related_name="grades", on_delete=models.CASCADE)
value = models.IntegerField()
Given the Fixtures
+------+------------------------+
| name | subject | grade_value |
+------+----------+-------------+
| beth | math | 100 |
| beth | history | 100 |
| beth | science | 100 |
| mark | math | 90 |
| mark | history | 90 |
| mark | science | 90 |
| mike | math | 90 |
| mike | history | 80 |
| mike | science | 80 |
+------+----------+-------------+
Desired Outcome
I want to render a list of students, but only include math and history grades.
For Example, maybe I want a list of students, but only include a subset of their grades:
GET students/?subjects=math,history
Which are filtered might be provided in the request, or hard coded.
If possible we can leave this outside the scope of this question, and assume the filtering parameters are fixed to math and history.
{
"students": [
{
"name": "beth",
"grades": [
{"subject": "math", "grade": 100 },
{"subject": "history", "grade": 100 },
// Exclude one or more grades - eg.
// science grade not included
]
},
...
]
}
Attempted Solution
Simple Filter
Just filtering. I guess this filters all students which have a grade with subjects in list, which is all.
queryset = Students.objects.all()\
.prefetch_related("grades")\
.filter(grades__subject__in=["math", "history"])
)
# all grades for each student eg.
...
"grades": [
{"subject": "math", "grade": 100 },
{"subject": "history", "grade": 100 },
{"subject": "science", "grade": 100 },
]
...
Subquery
I don't have a great grasp on how subqueries work, but using some examples I had I tried:
subjects = Subject.objects.filter(
name__in=["math", "history"]
)
queryset = Students.objects.all()\
.prefetch_related("grades")\
.filter(grades__subject__name__in=Subquery(subjects.values("name")))
And another variation:
grades = Grades.objects.filter(
student_id=OuterRef("id"), subject__name__in=["math", "history"]
)
queryset = Students.objects.all()\
.prefetch_related("grades")\
.filter(grades__pk__in=Subquery(grades.values("pk)))
Both returned students with all grades.
Workaround Solution
This solutions filters grades using python. It works but I would rather get this working with querysets
# in view:
serializer = StundentSerializer(queryset, many=True)
response_data = serializer.data
for student in response_data:
student.grades = [g for g in students.grades if g["subject"] in ["math", "history"]]
...
# return response_data

You can use the Prefetch object: https://docs.djangoproject.com/en/3.0/ref/models/querysets/#django.db.models.Prefetch
For e.g.:
qs = Students.objects.all().prefetch_related(
Prefetch('grades', queryset=Grade.objects.filter(subject__title__in=["math", "history"])
)
qs[0].grades.all() will now only have math and history grades.
Optionally you can provide the to_attr='math_history_grades' argument to Prefetch, so then you'll access the grades by: qs[0].math_history_grades.all()

Move the headings from top of Cucumber's Data Table to side - Python

I am looking for the ways to change the headings of Cucumber's Data Table to the side. So it will make the feature file readable.
Ordinary way:
| Name | Email | Phone No. | ......... |
| John | i#g.net | 098765644 | ......... |
It can be a very wide data table and I would have to scroll back and forth.
Desired way:
| Name | John |
| Email | i#g.net |
| Phone No. | 098765444 |
.
.
.
There are a small number of examples in Java and Ruby. But I am working with Python.
I had tried many different things like numpy.transpose(), converting them to list. But it won't work because the Data Table's format is:
[<Row['Name','John'],...]

You can implement this behaviour quite simply yourself, here is my version:
def tabledict(table, defaults, aliases = {}):
"""
Converts a behave context.table to a dictionary.
Throws NotImplementedError if the table references an unknown key.
defaults should contain a dictionary with the (surprise) default values.
aliases makes it possible to map alternative names to the keys of the defaults.
All keys of the table will be converted to lowercase, you sould make sure that
you defaults and aliases dictionaries also use lowercase.
Example:
Given the book
| Property | Value |
| Title | The Tragedy of Man |
| Author | Madach, Imre |
| International Standard Book Number | 9631527395 |
defaults = { "title": "Untitled", "author": "Anonymous", "isbn": None, "publisher": None }
aliases = { "International Standard Book Number" : "isbn" }
givenBook = tabledict(context.table, defaults, aliases)
will give you:
givenBook == {
"title": "The Tragedy of Man",
"author": "Madach, Imre",
"isbn": 9631527395,
"publisher": None
}
"""
initParams = defaults.copy()
validKeys = aliases.keys()[:] + defaults.keys()[:]
for row in table:
name, value = row[0].lower(), row[1]
if not name in validKeys:
raise NotImplementedError(u'%s property is not supported.'%name)
if name in aliases: name = aliases[name]
initParams[name] = value
return initParams

This doesn't look like it's related to numpy.
pivoting a list of list is often done with zip(*the_list)
This will return a pivoted behave table
from behave.model import Table
class TurnTable(unittest.TestCase):
"""
"""
def test_transpose(self):
table = Table(
['Name', 'John', 'Mary'],
rows=[
['Email', "john#example.com", "mary#example.com"],
['Phone', "0123456789", "9876543210"],
])
aggregate = [table.headings[:]]
aggregate.extend(table.rows)
pivoted = list(zip(*aggregate))
self.assertListEqual(pivoted,
[('Name', 'Email', 'Phone'),
('John', 'john#example.com', '0123456789'),
('Mary', 'mary#example.com', '9876543210')])
pivoted_table = Table(
pivoted[0],
rows=pivoted[1:])
mary = pivoted_table.rows[1]
self.assertEqual(mary['Name'], 'Mary')
self.assertEqual(mary['Phone'], '9876543210')
you can also have a look at https://pypi.python.org/pypi/pivottable

a search box with options relevant for a particular item that is selected django

Am having a search box, that i can use to search for items in django database and prints the results out at the template as a table.
Presently the code is working but i intend customizing the search, to accommodate a select tag so that whatever the user typed and and selected should query the database and fetch the outcome or result.
i tried but it not working
q is gotten from text input while x is gotten from select tag
here is my code:
def search_form(request):
data = OrderItem.objects.order_by().values_list('certification_type', flat=True).distinct()
print data
if 'q' in request.GET and request.GET['q']:
q = request.GET['q']
print q
x = request.GET['certification_type']
print x
items = OrderItem.objects.filter(Q(order_type__iexact=x and order_type__iexact=q) | Q(certification_type__iexact=x and certification_type__iexact=q) | Q(item__iexact=x and item__iexact=q)
| Q(certification_no__iexact=x and certification_no__iexact=q) | Q(client__user__email__iexact=x and client__user__email__iexact=q) | Q(client__phone_number__iexact=x and client__phone_number__iexact=q)
| Q(created_on__icontains=x and created_on__icontains=q))
for x in items:
for q in items:
print items
return render(request, 'i/search_results.html', {'items':items, 'q_query':q, 'x_query':x})
else:
return render(request, 'i/search_form.html', {'error':True, "data":data})

I think you are misunderstanding how lookups with Q objects work.
Assuming you want to filter OrderItem objects by
occurrence of the search string q in any attribute
certification_type gotten from the select tag x
you could do the following:
items = OrderItem.objects.filter(
certification_type=x,
Q(order_type__iexact=q) |
Q(item__iexact=q) |
Q(certification_no__iexact=q) |
Q(client__user__email__iexact=q) |
Q(client__phone_number__iexact=q) |
Q(created_on__icontains=q)
)

How to write a query with many ANDs and ORs with sum in Python?

I'm trying to write the following query in Django/Python:
in this query, I don't have a simple or, but I have many ands also and a sum
SELECT sum(value) FROM myapp_category
WHERE (name='difficulty' AND key='hard')
OR (name= 'success' AND key='yes')
OR (name= 'alternative' AND key='yes')
OR (name= 'processing' AND key='good')
OR (name= 'personal_touch' AND key='yes') `
and here's my model:
class Category(models.Model):
name = models.CharField(max_length=50)
key = models.CharField(max_length=30)
value = models.FloatField(blank=True, default=0)
def __str__(self):
return self.name.encode('utf_8') + "_" + self.key.encode('utf_8')
and I don't want to use the raw sql, so what can I use for this ?
Update Answer:
Thanks for your answers, this is the complete answer:
sum = Category.objects.filter(Q(name='difficulty',key=evaluation.difficulty) |
Q(name='nogos',key=evaluation.nogos) |
Q(name='success',key=evaluation.success) |
Q(name='alternative',key=evaluation.alternative) |
Q(name='processing',key=evaluation.processing) |
Q(name='personal_touch',key=evaluation.personal_touch))
.aggregate(result=Sum('value'))
score = float(sum['result'])

Try this;
from django.db.models import Q, Sum
Category.objects.filter(Q(name='difficulty',key='hard') | Q(name='success',key='yes') | Q(name='alternative',key='yes') | Q(name='processing',key='good') | Q(name='personal_touch',key='yes')).aggregate(Sum('value'))

UPD
from django.db.models import Q
results = Category.objects.filter(
Q(name="difficulty", key="hard") | Q(name= "success", key="yes") |
Q(name="alternative", key="yes")|Q(name="processing" AND key="good") |
Q(name="personal_touch",key="yes"))

Not really a solution but maybe input for other ideas:
SELECT sum(value) FROM myapp_category
WHERE name+"/"+key IN (
'difficulty/hard',
'success/yes',
'alternative/yes',
'processing/good',
'personal_touch/yes'
)
Problems:
Also lists cannot be provided using parameters
The query is slower because indexes cannot be used

Can I append twice the same object to an InstrumentedList in SQLAlchemy?

I have a pretty simple N:M relationship in SqlAlchemy 0.6.6. I have a class "attractLoop" that can contain a bunch of Media (Images or Videos). I need to have a list in which the same Media (let's say image) can be appended twice. The relationship is as follows:
The media is a base class with most of the attributes Images and Videos will share.
class BaseMedia(BaseClass.BaseClass, declarativeBase):
__tablename__ = "base_media"
_polymorphicIdentity = Column("polymorphic_identity", String(20), key="polymorphicIdentity")
__mapper_args__ = {
'polymorphic_on': _polymorphicIdentity,
'polymorphic_identity': None
}
_name = Column("name", String(50))
_type = Column("type", String(50))
_size = Column("size", Integer)
_lastModified = Column("last_modified", DateTime, key="lastModified")
_url = Column("url", String(512))
_thumbnailFile = Column("thumbnail_file", String(512), key="thumbnailFile")
_md5Hash = Column("md5_hash", LargeBinary(32), key="md5Hash")
Then the class who is going to use these "media" things:
class TestSqlAlchemyList(BaseClass.BaseClass, declarativeBase):
__tablename__ = "tests"
_mediaItems = relationship("BaseMedia",
secondary=intermediate_test_to_media,
primaryjoin="tests.c.id == intermediate_test_to_media.c.testId",
secondaryjoin="base_media.c.id == intermediate_test_to_media.c.baseMediaId",
collection_class=list,
uselist=True
)
def __init__(self):
super(TestSqlAlchemyList, self).__init__()
self.mediaItems = list()
def getMediaItems(self):
return self._mediaItems
def setMediaItems(self, mediaItems):
if mediaItems:
self._mediaItems = mediaItems
else:
self._mediaItems = list()
def addMediaItem(self, mediaItem):
self.mediaItems.append(mediaItem)
#log.debug("::addMediaItem > Added media item %s to %s. Now length is %d (contains: %s)" % (mediaItem.id, self.id, len(self.mediaItems), list(item.id for item in self.mediaItems)))
def addMediaItemById(self, mediaItemId):
mediaItem = backlib.media.BaseMediaManager.BaseMediaManager.getById(int(mediaItemId))
if mediaItem:
if mediaItem.validityCheck():
self.addMediaItem(mediaItem)
else:
raise TypeError("Media item with id %s didn't pass the validity check" % mediaItemId)
else:
raise KeyError("Media Item with id %s not found" % mediaItem)
mediaItems = synonym('_mediaItems', descriptor=property(getMediaItems, setMediaItems))
And the intermediate class to link both of the tables:
intermediate_test_to_media = Table(
"intermediate_test_to_media",
Database.Base.metadata,
Column("id", Integer, primary_key=True),
Column("test_id", Integer, ForeignKey("tests.id"), key="testId"),
Column("base_media_id", Integer, ForeignKey("base_media.id"), key="baseMediaId")
)
When I append the same Media object (instance) twice to one instances of that TestSqlAlchemyList, it appends two correctly, but when I retrieve the TestSqlAlchemyList instance from the database, I only get one. It seems to be behaving more like a set.
The intermediate table has properly all the information, so the insertion seems to be working fine. Is when I try to load the list from the database when I don't get all the items I had inserted.
mysql> SELECT * FROM intermediate_test_to_media;
+----+---------+---------------+
| id | test_id | base_media_id |
+----+---------+---------------+
| 1 | 1 | 1 |
| 2 | 1 | 1 |
| 3 | 1 | 2 |
| 4 | 1 | 2 |
| 5 | 1 | 1 |
| 6 | 1 | 1 |
| 7 | 2 | 1 |
| 8 | 2 | 1 |
| 9 | 2 | 1 |
| 10 | 2 | 2 |
| 11 | 2 | 1 |
| 12 | 2 | 1 |
As you can see, the "test" instance with id=1 should have the media [1, 1, 2, 2, 1, 1]. Well, it doesn't. When I load it from the DB, it only has the media [1, 2]
I have tried to set any parameter in the relationship that could possibly smell to list... uselist, collection_class = list... Nothing...
You will see that the classes inherit from a BaseClass. That's just a class that isn't actually mapped to any table but contains a numeric field ("id") that will be the primary key for every class and a bunch of other methods useful for the rest of the classes in my system (toJSON, toXML...). Just in case, I'm attaching an excerpt of it:
class BaseClass(object):
_id = Column("id", Integer, primary_key=True, key="id")
def __hash__(self):
return int(self.id)
def setId(self, id):
try:
self._id = int(id)
except TypeError:
self._id = None
def getId(self):
return self._id
#declared_attr
def id(cls):
return synonym('_id', descriptor=property(cls.getId, cls.setId))
If anyone can give me a push, I'll appreciate it a lot. Thank you. And sorry for the huge post... I don't really know how to explain better.

I think what you want is a described in the documentation as "Extra Fields in Many-to-Many Relationships". Rather than storing a unique row in the database foreach "link", between attractLoop and Media, you would store a single association and specify (as a part of the link object model) how many times it is referenced and/or in which location(s) in the final list the media should appear. This is a different paradigm from where you started, so it'll certainly require some re-coding, but I think it addresses your issue. You would likely need to use a property to redefine how to add or remove Media from the attactLoop.