Confusing [...] List in Python: What is it? - python

So I was writing up a simple binary tree in Python and came across [...]
I don't believe this to be related to the Ellipsis object, more it seems to have something to do with an infinity loop (due to Python's shallow copy?). The source of this infinity loop and why it doesn't get expanded while expanding when accessed is something I'm completely lost to, however
>>> a
[[[[[], [], 8, 3], [[], [], 3, 2], 6, 3], [], 1, 4], [[], [], -4, 2], 0, 0]
>>> Keys(a)#With a+b
[0, 1, 6, 8, 3, -4]
>>> Keys(a)#With [a,b]
[8, [...], [...], 3, [...], [...], 6, [...], [...], 1, [...], [...], -4, [...], [...], 0, [...], [...]]
>>> Keys(a)[1]#??
[8, [...], [...], 3, [...], [...], 6, [...], [...], 1, [...], [...], -4, [...], [...], 0, [...], [...], 8, [...], [...], 3, [...], [...], 6, [...], [...], 1, [...], [...], -4, [...], [...], 0, [...], [...]]
Version using a+b
def Keys(x,y=[]):
if len(x):y+=[x[2]]+Keys(x[0],y)+Keys(x[1],y)#Though it seems I was using y=y[:]+, this actually outputs an ugly mess
return y
version using [a,b]
def Keys(x,y=[]):
if len(x):y+=[x[2],Keys(x[0],y),Keys(x[1],y)]
return y
So what exactly is [...]?

It can also appear if you have a circular structure with a list pointing to itself. Like this:
>>> a = [1,2]
>>> a.append(a)
>>> a
[1, 2, [...]]
>>>
Since python can't print out the structure (it would be an infinite loop) it uses the ellipsis to show that there is recursion in the structure.
I'm not quite sure if the question was what what going on or how to fix it, but I'll try to correct the functions above.
In both of them, you first make two recursive calls, which add data to the list y, and then AGAIN append the returned data to y. This means the same data will be present several times in the result.
Either just collect all the data without adding to any y, with something like
return [x[2]]+keys(x[0])+keys(x[1])
or just do the appending in the calls, with something like
y += [x[2]]
keys(x[0], y) #Add left children to y...
keys(x[1], y) #Add right children to y...
return y
(Of course, both these snippets need handling for empty lists etc)
#Abgan also noted that you really don't want y=[] in the initializer.

I believe, that your 'tree' contains itself, therefore it contains cycles.
Try this code:
a = [1,2,3,4]
print a
a.append(a)
print a
The first print outputs:
[1,2,3,4]
while the second:
[1,2,3,4, [...]]
The reason is using
def Keys(x,y=[]):
This is wrong and evil. List is a mutable object, and when used as a default parameter, it is preserved between function calls.
So each y += "anything" operation adds to the same list (in all function calls, and since the function is recursive...)
See the Effbot or Devshed for more details on mutable objects passed as default values for functions.

I don't understand your code above, but the [...] I think is the Python interpreter skipping infinite data structures. For example:
>>> a = [0, 1]
>>> a[0] = a
>>> a
[[...], 1]
It looks like your tree structure is becoming looped.
The answers about slice objects are beside the point.

I don't believe this to be related to the Ellipsis object, more it seems to have something to do with an infinity loop (due to Python's shallow copy?). The source of this infinity loop and why it doesn't get expanded while expanding when accessed is something I'm completely lost to, however
Look at the following code:
>>> a = [0]
>>> a.append(a)
>>> print a
[0, [...]]
How is Python supposed to print a? It is a list that contains a zero and a reference to itself. Hence it is a list that contains a zero and a reference to a list
[0, [...]]
which in turn contains a zero and a reference to a list
[0, [0, [...]]]
which in turn contains a zero and a reference to a list,
and so on, recursively:
[0, [0, [0, [...]]]]
[0, [0, [0, [0, [...]]]]]
[0, [0, [0, [0, [0, [...]]]]]]
...
There is nothing wrong with the recursive data structure itself. The only problem is that it cannot be displayed, for this would imply an infinite recursion. Hence Python stops at the first recursion step and deals with the infinity issue printing only the ellipsis, as was pointed out in previous answers.

If you would have used a PrettyPrinter, the output would had been self explanatory
>>> l = [1,2,3,4]
>>> l[0]=l
>>> l
[[...], 2, 3, 4]
>>> pp = pprint.PrettyPrinter(indent = 4)
>>> pp.pprint(l)
[<Recursion on list with id=70327632>, 2, 3, 4]
>>> id(l)
70327632
In other words its something like

EDIT: As mentioned above, this isn't the Ellipsis object, but the result of a looped list. I jumped the gun here. Knowing about the Ellipsis object is a good bit of back shelf knowledge should you find an Ellipsis in some actual code, rather than the output.
The Ellipsis object in Python is used for extended slice notation. It's not used in current Python core libraries, but is available for developers to define in their own libraries. For example, NumPy (or SciPy) use this as part of their array object. You'll need to look at the documentation for tree() to know exactly how Ellipsis behaves in this object.
From Python documentation:
3.11.8 The Ellipsis Object
This object is used by extended slice
notation (see the Python Reference
Manual). It supports no special
operations. There is exactly one
ellipsis object, named Ellipsis (a
built-in name).
It is written as Ellipsis.

Ok, so in points:
You're creating infinite data
structure: def Keys(x,y=[]) will use the same 'y' in
each call. This just isn't correct.
The print statement, however, is clever enough not to print an infinite data, but to mark self-reference with a [...] (known as Ellipsis)
The Python will allow you to address such structure correctly, so you can write a.keys()[1][1][1] and so on. Why shouldn't you?
The y = y[:] statement simply copies the list y. Can be done more soundly with y = list(y)
Try using the following code:
def Keys(x,y=None):
if y is None:
y = []
if len(x):
y += [x[2], Keys(x[0],y), Keys(x[1],y)]
return y
But still I guess that it can bite you. You're still using the same variable y (I mean the same object) in three places in one expression:
y += [x[2], Keys(x[0], y), Keys(x[1], y)]
Is that what you really want to achieve?
Or maybe you should try:
def mKeys(x,y=None):
if y is None:
y = []
if len(x):
z = [x[2], mKeys(x[0], y), mKeys(x[1],y)]
return z
return []

For the difference between the two versions of the function Keys, note the following difference:
y+=[x[2]]+Keys(x[0],y)+Keys(x[1],y)
The right side value in this statement is a list which contains x[2], plus the ELEMENTS OF Keys(x[0],y) and the ELEMENTS OF Keys(x[1],y)
y+=[x[2],Keys(x[0],y),Keys(x[1],y)]
The right side value in this statement is a list which contains x[2], plus the LIST Keys(x[2],y) and the LIST Keys(x[1],y).
So the version using [a,b] will causing y contains itself as its elements.
Some other notes:
Since in python, the default value object is created once when the function is defined, the first version will not work like the example shows. It will contain multiple copy of some keys. It's hard to explain in short, but you can get some idea by printing the values of x, y on each call of Keys.
This is confirmed by running the function on my machine with python 2.5.2.
Also because the default value is defined only once at function definition time, even the function works correct for the first time, it will not work when calling with a different a, since the keys in the first binary tree will remain in y.
You can see this by calling Keys(a) twice, or calling it on two different lists.
The second parameter is not required for this problem. The function can be like this:
def Keys(a):
if a = []:
return []
else:
return [a[2]]+Keys(a[0])+Keys(a[1])
Defining a recursive function basically contains two part, solve subproblems and combined the results. In your code, the combining results part is repeated twice: one by accumulating them in y, one by adding the list together.

The issue is because one of the list element is referencing the list itself. So if an attempt to print all the elements is made then it would never end.
Illustration:
x = range(3)
x.append(x)
x[3][3][3][3][3][0] = 5
print x
Output:
[5, 1, 2, [...]]
x[3] is a referring to x itself. Same goes for x[3][3].
This can be visualized better
here

Related

Python: Having Trouble trying to use Cycle Function from Sympy on a list

I am trying the use the cycle function from Sympy to simplify down a list like so.
from sympy.combinatorics import Permutation, Cycle
Cycle(1,2,3)(3,4,5)(7)
And the output should be...
Cycle(1, 2, 4, 5, 3)(7)
However, when I try using...
a_list = [[1,2,3,4],[4,5,7],[3,4,2]]
b = Cycle(a_list)
print(b)
I get this error
'tuple' object is not callable
I know that I am inputting the wrong kind of variable into cycle, but could someone tell me what I can do with Cycle. It is a function that does exactly what I need, I just need to find a way to convert a list into a cycle friendly type. Thanks for your help.
I think you're looking for something like this:
a_list = [[1,2,3,4],[4,5,7],[3,4,2]]
b = Cycle()
for i in a_list:
b = b(*tuple(i))
print(b)
To convert to a list, try b.list().
Explanation
tuple(i) converts [1,2,3,4] to (1,2,3,4)
Say you have a function foo. Running foo(1,2,3,4) is the same as running foo(*(1,2,3,4))
A simpler example:
a_list = [[1,2], [3,4]]
b = Cycle()
On the first iteration (i = [1,2]), calling b(*tuple(i)) is the same as calling b(1,2) which, because b = Cycle(), is really Cycle()(1,2) which is the same as Cycle(1,2) according to the docs.
On the second iteration (i = [3,4]), calling b(*tuple(i)) is really b(3,4) which is Cycle(1,2)(3,4)
Hopefully, that example makes some sense. It's a little confusing because there are so many parentheses. If you're still confused, you might want to run through the code step by step (maybe with a debugger) to help understand what happens.
Cycle (as described in the docstring) provides some subtle advantages to entry, but if all elements are present in the list of cycles that you have shown, then simply passing them to Permutation should suffice.
Remember that order matters:
>>> Cycle(1,3)(3,2)
Cycle(1, 2, 3)
>>> Cycle(2,3)(1,3)
Cycle(1, 3, 2)
>>> p = Permutation([[1,3],[3,2]])
>>> Cycle(p)
Cycle(1, 2, 3)
>>> p.list() == _.list() == [0, 2, 3, 1]
Note that the Permutation will allow you to explicitly ask for array_form or cyclic_form whereas Cycle only allows the list() method:
>>> p.array_form
[0, 2, 3, 1]
>>> p.cyclic_form
[[1, 2, 3]]

How to fix method in python that returns a 2d array with empty array elements?

The following code implements a backtracking algorithm to find all the possible permutations of a given array of numbers and the record variable stores the permutation when the code reaches base case. The code seems to run accordingly, that is, the record variable gets filled up with valid permutations, but for some reason when the method finishes the method returns a two-dimensional array whose elements are empty.
I tried declaring record as a tuple or a dictionary and tried using global and nonlocal variables, but it none of it worked.
def permute(arr):
record = []
def createPermutations(currentArr, optionArr):
if len(optionArr) == 0:
if len(currentArr) != 0: record.append(currentArr)
else: pass
print(record)
else:
for num in range(len(optionArr)):
currentArr.append(optionArr[num])
option = optionArr[0:num] + optionArr[num+1::]
createPermutations(currentArr, option)
currentArr.pop()
createPermutations([], arr)
return record
print(permute([1,2,3]))
The expect result should be [[1, 2, 3], [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 1, 2], [3, 2, 1]], but instead I got [[], [], [], [], [], []].
With recursive functions, you should pass a copy of the current array, rather than having all of those currentArr.pop() mutating the same array.
Replace
createPermutations(currentArr, option)
by
createPermutations(currentArr[:], option)
Finally, as a learning exercise for recursion, something like this is fine, but if you need permutations for a practical programming problem, use itertools:
print([list(p) for p in itertools.permutations([1,2,3])])
I would accept John Coleman's answer as it is the correct way to solve your issue and resolves other bugs that you run into as a result.
The reason you run into this issue because python is pass-by-object-reference, in which copies of lists are not passed in but the actual list itself. What this leads to is another issue in your code; in which you would get [[3, 2, 1], [3, 2, 1], [3, 2, 1], [3, 2, 1], [3, 2, 1], [3, 2, 1]] as your output when you print(record).
Why this happens is that when you call record.append(currentArr), it actually points to the same object reference as all the other times you call record.append(currentArr). Thus you will end up with 6 copies of the same array (in this case currentArr) at the end because all your appends point to the same array. A 2d list is just a list of pointers to other lists.
Now that you understand this, it is easier to understand why you get [[],[],[],[],[],[]] as your final output. Because you add to and then pop from currentArr over here currentArr.append(optionArr[num])
and over here
currentArr.pop() to return it back to normal,
your final version of currentArr will be what you passed in, i.e. [].
Since result is a 2d array of 6 currentArrs, you will get [[],[],[],[],[],[]] as your returned value.
This may help you better how it all works, since it has diagrams as well: https://robertheaton.com/2014/02/09/pythons-pass-by-object-reference-as-explained-by-philip-k-dick/

Python mutation of lists in tuples

After learning about lists and box-and-pointer diagrams, I decided to create random stuff for myself and test out my knowledge. I am going to use the words shallow copy and suspected shallow copies as I'm not really sure whether they are correct by definition. My queries are in the reasons provide for the behaviour of such code, please tell me whether I'm thinking soundly.
Code A
from copy import *
x=[1,[2,[3,[4]]]] #normal copy/hardcopy
a=x
v=list(x) #suspected shallow copy
y=x.copy() #shallow copy
z=deepcopy(x) #theoretical deep copy
w=x[:] #suspected shallow copy
def test():
print("Original:",x)
print("hardcopy:",a)
print("suspected shallow copy",v)
print("shallow copy",y)
print("deep copy:",z)
print("suspected shallow copy",w)
x[1]=x[1]+[4]
test()
Output A:
Original: [1, [2, [3, [4]], 4]]
hardcopy: [1, [2, [3, [4]], 4]]
suspected shallow copy [1, [2, [3, [4]]]]
shallow copy [1, [2, [3, [4]]]]
deep copy: [1, [2, [3, [4]]]]
suspected shallow copy [1, [2, [3, [4]]]]
Code B
a=(1,2,[1,2,3])
def shallow_copy(x):
tup=()
for i in x:
tup+=(i,)
return tup
def hardcopy(x):
return x
b=hardcopy(a)
c=shallow_copy(a)
a[2]+=[3]
Output B:
I see TypeError in IDLE here, but the mutation of the list element is still done, and across ALL a,b,c
Continuation from output B:
a[2][0]=a[2][0]+99
a,b,c
Output C:
((1, 2, [100, 2, 3, 3]), (1, 2, [100, 2, 3, 3]), (1, 2, [100, 2, 3, 3]))
Code D:
a=[1,2,(1,2,3)]
def shallow_copy(x):
tup=[]
for i in x:
tup+=[i]
return tup
def hardcopy(x):
return x
b=hardcopy(a)
c=shallow_copy(a)
d=a.copy()
a[2]=a[2]+(4,)
a,b,c,d
Output D:
[1, 2, (1, 2, 3, 4)], [1, 2, (1, 2, 3, 4)],
[1, 2, (1, 2, 3)], [1, 2, (1, 2, 3)]
From Output A, we observe the following:
1)For lists which have shallow copies, doing x[1]=x[1]+[4] does not affect the shallow copies. My reasons for the above could be
a) = followed by + does __add__ instead of __iadd__(which is +=), and doing __add__ should not modify the object, only changing the value for one pointer(x and its hardcopy in this case)
This is further supported in Output B but somehow contradicted in Output C, could be partly due to reason (b) below, but can't be too sure.
b) We executed this in the first layer(only 1 slice operator), maybe there's some kind of rule which prevents these elements from being modified.
This is supported by both Output B and Output C, though Output B might be argued to be in the first layer, think of it as increasing the elements in the 2nd layer, and it fits the above observation.
2)What is the reason why the TypeError appeared in Output B, but is still executed? I know that whether an Exception might be triggered is based on the final sequence you are actually changing(the list in this case), but why is there still TypeError: 'tuple' object does not support item assignment ?
I have presented my views for the above questions. I appreciate any thoughts(theoretical solutions preferably) on this question as I'm still relatively new to programming.
To answer question 1, which looks complex but whose answer is probably quite simple:
when you have a another name referencing the original object, you will see the changes in the original. Those changes will not reflect in other copies (being those either shallow or deep) if(!) you change the objects using the form x[1] = x[1] + [4]. This is because you are assigning a new object into x[1], instead of making an in-place change like in x[1].append(4).
You can check that with the id() function.
To answer your question 2, and adapted from the official docs:
let's make
a = (['hello'],)
then
a[0] += [' world']
this is the same as
a[0] = operator.iadd(a[0],[' world'])
The iadd changes the list in place, but then the assignment fails because you can't assign to a tuple (immutable type) index.
If you make
a[0] = a[0] + [' world']
the concatenation goes into a new list object, then the assignment to the tuple index fails too. But the new object gets lost. a[0] wasn't changed in place.
To clarify OP's comment, directly from the docs in here it says that
Many operations have an “in-place” version. Listed below are functions providing a more primitive access to in-place operators than the usual syntax does; for example, the statement x += y is equivalent to x = operator.iadd(x, y). Another way to put it is to say that z = operator.iadd(x, y) is equivalent to the compound statement z = x; z += y.
In those examples, note that when an in-place method is called, the
computation and assignment are performed in two separate steps. The
in-place functions listed below only do the first step, calling the
in-place method. The second step, assignment, is not handled.
As for your Output D:
Writing
b = hardcopy(a)
does nothing more than writing
b = a
really, b is a new name referencing the same object that a references.
This is because a is mutable and so a reference pointing to the original object is passed into local function name x. Returning x just returns the same reference into b.
That's why you see further changes in a reflected in b. Again you make a[2] a new different object tuple by assignment, so now a[2] and b[2] reference a new tuple (1,2,3,4), while c and d still reference the old tuple object. And now because they are tuples you can't change them in place, like lists.
As for the term "hardcopy", I wouldn't use it. It doesn't appear even once in official docs, and the mentions in Python SO questions beside this one, appear in other contexts. And it is ambiguous (contrary to "shallow" and "deep" which give a good clue for their meaning). I would think exactly the opposite (an object copy) for the term "hardcopy" you describe (an additional name/reference/pointer to the same object). Of course there are eventually many ways to say the same thing. We say "copy" because its shorter, and for immutables it doesn't matter if the copy happens or not (you can't change them anyway). For mutables saying "copy" usually means "shallow copy", because you have to "go further" in your code if you want a "deep copy".

How to simultaneously iterate and modify list, set, etc?

In my program I have many lines where I need to both iterate over a something and modify it in that same for loop.
However, I know that modifying the thing over which you're iterating is bad because it may - probably will - result in an undesired result.
So I've been doing something like this:
for el_idx, el in enumerate(theList):
if theList[el_idx].IsSomething() is True:
theList[el_idx].SetIt(False)
Is this the best way to do this?
This is a conceptual misunderstanding.
It is dangerous to modify the list itself from within the iteration, because of the way Python translates the loop to lower level code. This can cause unexpected side effects during the iteration, there's a good example here :
https://unspecified.wordpress.com/2009/02/12/thou-shalt-not-modify-a-list-during-iteration/
But modifying mutable objects stored in the list is acceptable, and common practice.
I suspect that you're thinking that because the list is made up of those objects, that modifying those objects modifies the list. This is understandable - it's just not how it's normally thought of. If it helps, consider that the list only really contains references to those objects. When you modify the objects within the loop - you are merely using the list to modify the objects, not modifying the list itself.
What you should not do is add or remove items from the list during the iteration.
Your problem seems to be unclear to me. But if we talk about harmful of modifying list during a for loop iteration in Python. I can think about two scenarios.
First, You modify some elements in list that suppose to be used on the next round of computation as its original value.
e.g. You want to write a program that have such inputs and outputs like these.
Input:
[1, 2, 3, 4]
Expected output:
[1, 3, 6, 10] #[1, 1 + 2, 1 + 2 + 3, 1 + 2 + 3 + 4]
But...you write a code in this way:
#!/usr/bin/env python
mylist = [1, 2, 3, 4]
for idx, n in enumerate(mylist):
mylist[idx] = sum(mylist[:idx + 1])
print mylist
Result is:
[1, 3, 7, 15] # undesired result
Second, you make some change on size of list during a for loop iteration.
e.g. From python-delete-all-entries-of-a-value-in-list:
>>> s=[1,4,1,4,1,4,1,1,0,1]
>>> for i in s:
... if i ==1: s.remove(i)
...
>>> s
[4, 4, 4, 0, 1]
The example shows the undesired result that raised from side-effect of changing size in list. This obviously shows you that for each loop in Python can not handle list with dynamic size in a proper way. Below, I show you some simple way to overcome this problem:
#!/usr/bin/env python
s=[1, 4, 1, 4, 1, 4, 1, 1, 0, 1]
list_size=len(s)
i=0
while i!=list_size:
if s[i]==1:
del s[i]
list_size=len(s)
else:
i=i + 1
print s
Result:
[4, 4, 4, 0]
Conclusion: It's definitely not harmful to modify any elements in list during a loop iteration, if you don't 1) make change on size of list 2) make some side-effect of computation by your own.
you could get index first
idx = [ el_idx for el_idx, el in enumerate(theList) if el.IsSomething() ]
[ theList[i].SetIt(False) for i in idx ]

Wrapping around a list as a slice operation

Consider the following simple python code
>>> L = range(3)
>>> L
[0, 1, 2]
We can take slices of this array as follows:
>>> L[1:3]
[1, 2]
Is there any way to wrap around the above array by shifting to the left
[1, 2, 0]
by simply using slice operations?
Rotate left n elements (or right for negative n):
L = L[n:] + L[:n]
Note that collections.deque has support for rotations. It might be better to use that instead of lists.
Left:
L[:1], L[1:] = L[-1:], L[:-1]
Right:
L[-1:], L[:-1] = L[:1], L[1:]
To my mind, there's no way, unless you agree to cut and concatenate lists as shown above.
To make the wrapping you describe you need to alter both starting and finishing index.
A positive starting index cuts away some of initial items.
A negative starting index gives you some of the tail items, cutting initial items again.
A positive finishing index cuts away some of the tail items.
A negative finishing index gives you some of the initial items, cutting tail items again.
No combination of these can provide the wrapping point where tail items are followed by initial items. So the entire thing can't be created.
Numerous workarounds exist. See answers above, see also itertools.islice and .chain for a no-copy sequential approach if sequential access is what you need (e.g. in a loop).
If you are not overly attached to the exact slicing syntax, you can write a function that produces the desired output including the wrapping behavior.
E.g., like this:
def wrapping_slice(lst, *args):
return [lst[i%len(lst)] for i in range(*args)]
Example output:
>>> L = range(3)
>>> wrapping_slice(L, 1, 4)
[1, 2, 0]
>>> wrapping_slice(L, -1, 4)
[2, 0, 1, 2, 0]
>>> wrapping_slice(L, -1, 4, 2)
[2, 1, 0]
Caveat: You can't use this on the left-hand side of a slice assignment.

Categories

Resources