Develop Reference

append to request.sessions[list] in Django

Something is bugging me.
I'm following along with this beginner tutorial for django (cs50) and at some point we receive a string back from a form submission and want to add it to a list:
https://www.youtube.com/watch?v=w8q0C-C1js4&list=PLhQjrBD2T380xvFSUmToMMzERZ3qB5Ueu&t=5777s
def add(request):
if 'tasklist' not in request.session:
request.session['tasklist'] = []
if request.method == 'POST':
form_data = NewTaskForm(request.POST)
if form_data.is_valid():
task = form_data.cleaned_data['task']
request.session['tasklist'] += [task]
return HttpResponseRedirect(reverse('tasks:index'))
I've checked the type of request.session['tasklist']and python shows it's a list.
The task variable is a string.
So why doesn't request.session['tasklist'].append(task) work properly? I can see it being added to the list via some print statements but then it is 'forgotten again' - it doesn't seem to be permanently added to the tasklist.
Why do we use this request.session['tasklist'] += [task] instead?
The only thing I could find is https://ogirardot.wordpress.com/2010/09/17/append-objects-in-request-session-in-django/ but that refers to a site that no longer exists.
The code works fine, but I'm trying to understand why you need to use a different operation and can't / shouldn't use the append method.
Thanks.

The reason why it does not work is because django does not see that you have changed anything in the session by using the append() method on a list that is in the session.
What you are doing here is essentially pulling out the reference to the list and making changes to it without the session backend knowing anything about it. An other way to explain:
The append() method is on the list itself not on the session object
When you call append() on the list you are only talking to the list and the list's parent (the session) has no idea what you guys are doing
When you however do an assignment on the session itself session['whatever'] = 'something' then it knows that something is up and changes are made
So the key here is that you need to operate on the session object directly if you want your changes to be updated automatically
Django only thinks it needs to save a changed session item if the item got reassigned to the session. See here: django session base code the __setitem__ method containing a self.modified = True statement.
The session['list'] += [new_element] adds a new list item (mutates the list stored in the session, so the list reference stays the same) and then gets it reassigned to the session again -> thus triggering first a __getitem__ call -> then your += / __iadd__ runs on the value read -> then a __setitem__ call is made (with the list ref. passed to it). You can see it in the django codebase that it marks the session after each __setitem__ call as modified.
The session['list'] = session['list'] + [new_item] mode of doing the same does create a new list every time it's run so its a bit less efficient, but you should not store hundreds of items in the session anyway. So you're probably fine. This also works exactly as above.
However if you use sub-keys in the session like session['list']['x'] = 'whatever' the session will not see itself as modified so you need to mark it as by request.session.modified = True

Short answer: It's about how Python chooses to implement the dict data structure.
Long answer:
Let's start by saying that request.session is a dictionary.
Quoting Django's documentation, "By default, Django only saves to the session database when the session has been modified – that is if any of its dictionary values have been assigned or deleted". Link
So, the problem is that the session database is not being modified by
request.session['tasklist'].append(task)
Seeing the related parts Django's Session base code (as posted by #Csaba K. in an answer), the variable self.modified is to be set True when setitem dunder method is called.
Now, at this step the problem seems like the setitem dunder method is not being called with request.session['tasklist'].append(task) but with request.session['tasklist'] += [task] it gets called. It is not due to if the reference of request.session['tasklist'] is changing or not as pointed out by another answer, because the reference to the underlying list remains the same.
To confirm, let's create a custom dictionary which extends the Python dict, and print something when setitem dunder method is called.
class MyDict(dict):
def __init__(self, globalVar):
super().__init__()
self.globalVar = globalVar
def __setitem__(self, key, value):
super().__setitem__(key, value)
print("Called Set item when: ", end="")
myDict = MyDict(0)
print("Creating Dict")
print("-----")
myDict["y"] = []
print("Adding a new key-value pair")
print("-----")
myDict["y"] += ["x"]
print(" using +=")
print("-----")
myDict["y"].append("x")
print("append")
print("-----")
myDict["y"].extend(["x"])
print("extend")
print("-----")
myDict["y"] = myDict["y"] + ["x"]
print(" using +",)
print("-----")
It prints:
Creating Dict
-----
Called Set item when: Adding a new key-value pair
-----
Called Set item when: using +=
-----
append
-----
extend
-----
Called Set item when: using +
-----
As we can see, setitem dunder method is called and in turn self.modified is set true only when adding a new key-value pair, or using += or using +, but not when initializing, appending or extending an iterable (in this case a list). Now, the operator + and += do very different things in Python, as explained in the other answer. += behaves more like the append method but in this case, I guess it's more about how Python chooses to implement the dict data structure rather than how +, += and append behave on lists.

I found this while doing some more searching:
https://code.djangoproject.com/wiki/NewbieMistakes
Scroll to 'Appending to a list in session doesn't work'
Again, it is a very dated entry but still seems to hold true.
Not completely satisfied because this does not answer the question as to 'why' this doesn't work, but at the very least confirms 'something's up' and you should probably still use the recommendations there.
(if anyone out there can actually explain this in a more verbose manner then I'd be happy to hear it)

How to change list of objects into list of dicts?

I am quite surprised at the behavior of this code, which is inside a function:
for user in full_details["users"]:
user = collections.defaultdict(lambda: False, user)
if user["comments"]:
user["comments"] = [comment.__dict__ for comment in user["comments"]]
print("just converted user comments to dict objects")
print(user["comments"])
print("printing full details")
print(full_details)
My understanding was that if I modified a dictionary or a list, that modification applied to the object and would remain. However, when I change user["comments"] for each user in full_details["users"] within my if I am not then seeing those same changes again reflected in full_details just immediately after. Why is that? I thought that whenever you create a new list and assign it to a passed-in parameter, that new list will persist outside the function.
My trouble is that the change made here does not persist:
user["comments"] = [comment.__dict__ for comment in user["comments"]]
Also, full_details is a default_dict:
full_details = collections.defaultdict(lambda: False, thread_details.__dict__)

You are assigning "user" twice. First in the for statement. Then in the first line of the body of the for loop, you are creating a new object and also assigning it to "user". At this point, you have lost your reference to the original object.

Python: Using a multidimensional multiprocessing.manager.list()

This might not be its intended use, but I would like to know how to use a multidimensional manager.list(). I can create on just fine, something like this:
from multiprocessing import manager
test = manager.list(manager.list())
however when ever I try to access the first element of the test list it returns the element's value and not its proxy object
test[0] # returns [] and not the proxy, since I think python is running __getitem__.
Is there anyway for me to get around this and use the manager.list() in this way?

The multiprocessing documentation has a note on this:
Note
Modifications to mutable values or items in dict and list proxies will
not be propagated through the manager, because the proxy has no way of
knowing when its values or items are modified. To modify such an item,
you can re-assign the modified object to the container proxy:
# create a list proxy and append a mutable object (a dictionary)
lproxy = manager.list()
lproxy.append({})
# now mutate the dictionary
d = lproxy[0]
d['a'] = 1
d['b'] = 2
# at this point, the changes to d are not yet synced, but by
# reassigning the dictionary, the proxy is notified of the change
lproxy[0] = d
So, the only way to use a multidimensional list is to actually reassign any changes you make to the second dimension of the list back to the top-level list, so instead of:
test[0][0] = 1
You do:
tmp = test[0]
tmp[0] = 1
test[0] = tmp
Not the most pleasant way to do things, but you can probably write some helper functions to make it a bit more tolerable.
Edit:
It seems the reason that you get a plain list back when you append a ListProxy to another ListProxy is because of how pickling Proxies works. BaseProxy.__reduce__ creates a RebuildProxy object, which what actually is used to unpickle the Proxy. RebuildProxy looks like this:
def RebuildProxy(func, token, serializer, kwds):
'''
Function used for unpickling proxy objects.
If possible the shared object is returned, or otherwise a proxy for it.
'''
server = getattr(process.current_process(), '_manager_server', None)
if server and server.address == token.address:
return server.id_to_obj[token.id][0]
else:
incref = (
kwds.pop('incref', True) and
not getattr(process.current_process(), '_inheriting', False)
)
return func(token, serializer, incref=incref, **kwds)
As the docstring says, if the unpickling is occuring inside a manager server, the actual shared object is created, rather than the Proxy to it. This is probably a bug, and there is actually one filed against this behavior already.

Using an object's method to sort a list of objects that contains the object

I have a problem.
I am trying to generate a list of objects of different types sorted by the object's "date". Each object has a method getDate which returns the date to sort by.
The UserProfile object contains a method recent_activity which should return this sorted list of objects.
However, see the line #items.append(self) in the below?
class UserProfile(models.Model):
...
def getDate(self):
return self.last_edited
...
def recent_activity(self):
followed = ...
sponsored = ...
...
items = []
#items.append(self) #<-- If this is in, the call to sorted doesn't work
for f in followed:
if (...):
items.append(f)
for s in sponsored:
if (...):
items.append(s)
for c in comments:
if (...):
items.append(c)
for u in updates:
if (...):
items.append(u)
for p in projects:
items.append(p)
#return items[0:7] #<-- If this is in, everything is fine!
items = sorted(items, key=lambda item: item.getDate(),reverse=True)
return items[0:7]
When I comment out the line items.append(self), the code as written works perfectly, returning a (sliced) list of objects of different types sorted by date (using the getDate() method).
However, when this line is included, the code works intermittently - and I mean that for some objects it works (as above), and for some it does not (at all - in fact it often simply returns None).
If I return the array BEFORE calling sorted, again, everything works fine. So the problem is associated with the call to sorted.
Looking forward to hearing what I'm doing wrong!
(Just to be clear, someone has down-rated this question, I have no idea why - if you think it's unclear, please take the time to comment so I can improve the question!).
Edit - Now fixed. Solution was having occasional getDate() calls returning datetime.date objects rather than datetime.datetime objects because of a database migration causing me to manually populate some of the fields with the wrong type of object. Bit surprised to see no exceptions appear, just silent failing.

Using pdb I managed to get to the root of the problem.
A couple of objects in my database had datetime.date objects stored in the field that was supposed to store datetime.datetime objects. The comparisons between these two types of object was failing during sorting, but failing silently.
I'm not sure why these comparisons in sorted did not generate an exception - in my view they should have.
Anyway, thanks for your suggestions.

l[:] performance problem

Recently I was debugging a code like following
def getList():
#query db and return a list
total_list = Model.objects.all()
result = list()
for item in total_list:
if item.attr1:
result.append(item)
return result
# in main code
org_list = getList()
list = orgList[:]#this line cause cpu problems.
if len(org_list)>0 and org_list[0].is_special:
my_item = org_list[0]
for i in list:
print_item(i)
doSomethingelse(list[0])
In order to simplify the code I change most of it but the main part is here.
In getList method we query db and get 20~30 rows. Then we create a python list from it and return it.
in main method we get org_list variable from getList method
and slice it with orgList[:]
and loop over this list and call spesific elements on it like list[0]
The problem here is that this code runs on a really busy server and unfortunaletty it uses most of the cpu and eventually locks our servers.
Problem here is the line that we slice list varibale with list[:]
if we dont do that and just use org_list variable instead our servers does not have a problem. Does anybosy have any idea why that might happen. is slicing uses alot of cpu or when we use a sliced list. does it uses alot of cpu?

The code that you are showing would run in the 0.1 microseconds that it would take to raise an exception:
org_list = getList()
list = orgList[:]#this line cause cpu problems.
orgList should be org_list. Show the minimal code that actually reproduces the problem.
Also, that kills the list built-in function. Don't do that.
Update Another thought: A common response to "My Django app runs slowly!" is "turn off the debug flag" ... evidently it doesn't free up memory in debug mode.
Update2 about """I found out that when you slice a list. it actually works as a view to that list and just call original list methods to get actual item.""" Where did you get that idea? That can't be the case with a plain old list! Have you redefined list somewhere?
In your getList function:
(1) put in print type(list)
(2) replace result = list() with result = [] and see whether the problem goes away.

list = org_list[:] makes a copy of org_list. Usually you see something like this when you need to modify the list but you want to keep the original around as well.
The code you are showing me doesn't seem like it actually modifies org_list, and if doSomethingelse(list[0]) doesn't modify list, I don't see why it's actually being copied in the first place. Even if it does modify list, as long as org_list isn't needed after those few lines of code, you could probably get away with just using org_list and not doing the slice copy.

I would like to know:
what is the type of total_list ? Please do print type(total_list)
what are the types of the elements of total_list ?
have you an idea of the sizes of these elements ?
what is the number of elements in the list returned by getList() ? 20-30 ?
Remarks:
-giving the keyword "list" as a name for a list isn't a good idea (in list = orgList[:] )
-your getList function can be written :
def getList():
#query db and return a list
total_list = Model.objects.all()
return [ item for item in total_list if item.attr1 ]

unfortunately actual code is property of the company I worked for so I cannot give it exactly like it is but here is some part of it (variable names are changed,but argument of mainTree method was really named list.)
mainObj = get_object_or_404(MainObj, code_name=my_code_name)
if (not mainObj.is_now()) and (not request.user.is_staff):
return HttpResponseRedirect("/")
main_tree = mainObj.main_tree()
objects = main_tree[:] # !!!
if objects.__len__() == 1 and not objects[0].has_special_page:
my_code_name = objects[0].code_name
...
...
...
mainObj.main_tree() is the following
def main_tree(self):
def append_obj_recursive(list, obj):
list.append(obj)
for c in self.obj_set.filter(parent_obj=obj):
append_obj_recursive(list, c)
return list
root_obj = self.get_or_create_root_obj();
top_level_objs = self.obj_set.filter(parent_obj=root_obj).order_by("weight")
objs = []
for tlc in top_level_objs:
append_obj_recursive(objs, tlc)
return objs
This is a really weird problem. slicing should not cause such a bizare problem. If I can find an answer I will post it here too.