Am using Python 2.7.
I have a list of lists like:
testList2 = [[u'462', u'San Germ\xe1n, PR'],[u'461', u'40341']]
I want to encode the strings in the list of lists like:
encodedList = [['462', 'San Germ\xc3\xa1n, PR'],['461', '40341']]
Tried to write a function to do this (did not work):
def testEncode(a):
for list in a:
return [x.encode('utf-8') for x in list]
I think that for the function to work, it needs to append each encoded list to the prior encoded list to generate an encoded list of lists. Not sure how to do this. If someone could explain how the function could be edited to do this, that would be awesome.
I tried the following which did not work either
def testEncode(a):
b = []
for list in a:
b.append([x.encode('utf-8') for x in list])
return b
Having realized that your first code is not actually a typographical error but a logical mistake, let me summarize my comments here. There are two problems (both related) in your approaches:
Problem with the first code: You are currently returning only the first sublist because you put the return in your for loop. Your input list contains sublists so you need to loop over them in a nested manner. One way is to do it as you are doing in your second approach. Another way is to use list comprehensions. Following is the list comprehension way where i will iterate through the sublists and x will iterate through the elements of your sublist i.
def testEncode(a):
return [[x.encode('utf-8') for x in i] for i in a]
Problem with the second code: In this attempt of yours, you have basically solved the problem of ignoring the sublists but you forgot to put your return statement outside the for loop. So before your nested for loop iterate through all the sublists, you prematurely return the result. Therefore, you only see the first sublist modified.
def testEncode(a):
b = []
for list in a:
b.append([x.encode('utf-8') for x in list])
return b # <-- Moved outside the for loop now
Related
I have the following list:
x = [(27.3703703703704, 2.5679012345679, 5.67901234567901,
6.97530864197531, 1.90123456790123, 0.740740740740741,
0.440136054421769, 0.867718446601942),
(25.2608695652174, 1.73913043478261, 6.07246376811594,
7.3768115942029, 1.57971014492754, 0.710144927536232,
0.4875, 0.710227272727273)]
I'm looking for a way to get the average of each of the lists nested within the main list, and create a new list of the averages. So in the case of the above list, the output would be something like:
[[26.315],[2.145],[5.87],etc...]
I would like to apply this formula regardless of the amount of lists nested within the main list.
I assume your list of tuples of one-element lists is looking for the sum of each unpacked element inside the tuple, and a list of those options. If that's not what you're looking for, this won't work.
result = [sum([sublst[0] for sublst in tup])/len(tup) for tup in x]
EDIT to match changed question
result = [sum(tup)/len(tup) for tup in x]
EDIT to match your even-further changed question
result = [[sum(tup)/len(tup)] for tup in x]
An easy way to acheive this is:
means = [] # Defines a new empty list
for sublist in x: # iterates over the tuples in your list
means.append([sum(sublist)/len(sublist)]) # Put the mean of the sublist in the means list
This will work no matter how many sublists are in your list.
I would advise you read a bit on list comprehensions:
https://docs.python.org/2/tutorial/datastructures.html
It looks like you're looking for the zip function:
[sum(l)/len(l) for l in zip(*x)]
zip combines a collection of tuples or lists pairwise, which looks like what you want for your averages. then you just use sum()/len() to compute the average of each pair.
*x notation means pass the list as though it were individual arguments, i.e. as if you called: zip(x[0], x[1], ..., x[len(x)-1])
r = [[sum(i)/len(i)] for i in x]
This question already has answers here:
How can I use list comprehensions to process a nested list?
(13 answers)
Closed 7 months ago.
I recently looked for a way to flatten a nested python list, like this: [[1,2,3],[4,5,6]], into this: [1,2,3,4,5,6].
Stackoverflow was helpful as ever and I found a post with this ingenious list comprehension:
l = [[1,2,3],[4,5,6]]
flattened_l = [item for sublist in l for item in sublist]
I thought I understood how list comprehensions work, but apparently I haven't got the faintest idea. What puzzles me most is that besides the comprehension above, this also runs (although it doesn't give the same result):
exactly_the_same_as_l = [item for item in sublist for sublist in l]
Can someone explain how python interprets these things? Based on the second comprension, I would expect that python interprets it back to front, but apparently that is not always the case. If it were, the first comprehension should throw an error, because 'sublist' does not exist. My mind is completely warped, help!
Let's take a look at your list comprehension then, but first let's start with list comprehension at it's easiest.
l = [1,2,3,4,5]
print [x for x in l] # prints [1, 2, 3, 4, 5]
You can look at this the same as a for loop structured like so:
for x in l:
print x
Now let's look at another one:
l = [1,2,3,4,5]
a = [x for x in l if x % 2 == 0]
print a # prints [2,4]
That is the exact same as this:
a = []
l = [1,2,3,4,5]
for x in l:
if x % 2 == 0:
a.append(x)
print a # prints [2,4]
Now let's take a look at the examples you provided.
l = [[1,2,3],[4,5,6]]
flattened_l = [item for sublist in l for item in sublist]
print flattened_l # prints [1,2,3,4,5,6]
For list comprehension start at the farthest to the left for loop and work your way in. The variable, item, in this case, is what will be added. It will produce this equivalent:
l = [[1,2,3],[4,5,6]]
flattened_l = []
for sublist in l:
for item in sublist:
flattened_l.append(item)
Now for the last one
exactly_the_same_as_l = [item for item in sublist for sublist in l]
Using the same knowledge we can create a for loop and see how it would behave:
for item in sublist:
for sublist in l:
exactly_the_same_as_l.append(item)
Now the only reason the above one works is because when flattened_l was created, it also created sublist. It is a scoping reason to why that did not throw an error. If you ran that without defining the flattened_l first, you would get a NameError
The for loops are evaluated from left to right. Any list comprehension can be re-written as a for loop, as follows:
l = [[1,2,3],[4,5,6]]
flattened_l = []
for sublist in l:
for item in sublist:
flattened_l.append(item)
The above is the correct code for flattening a list, whether you choose to write it concisely as a list comprehension, or in this extended version.
The second list comprehension you wrote will raise a NameError, as 'sublist' has not yet been defined. You can see this by writing the list comprehension as a for loop:
l = [[1,2,3],[4,5,6]]
flattened_l = []
for item in sublist:
for sublist in l:
flattened_l.append(item)
The only reason you didn't see the error when you ran your code was because you had previously defined sublist when implementing your first list comprehension.
For more information, you may want to check out Guido's tutorial on list comprehensions.
For the lazy dev that wants a quick answer:
>>> a = [[1,2], [3,4]]
>>> [i for g in a for i in g]
[1, 2, 3, 4]
While this approach definitely works for flattening lists, I wouldn't recommend it unless your sublists are known to be very small (1 or 2 elements each).
I've done a bit of profiling with timeit and found that this takes roughly 2-3 times longer than using a single loop and calling extend…
def flatten(l):
flattened = []
for sublist in l:
flattened.extend(sublist)
return flattened
While it's not as pretty, the speedup is significant. I suppose this works so well because extend can more efficiently copy the whole sublist at once instead of copying each element, one at a time. I would recommend using extend if you know your sublists are medium-to-large in size. The larger the sublist, the bigger the speedup.
One final caveat: obviously, this only holds true if you need to eagerly form this flattened list. Perhaps you'll be sorting it later, for example. If you're ultimately going to just loop through the list as-is, this will not be any better than using the nested loops approach outlined by others. But for that use case, you want to return a generator instead of a list for the added benefit of laziness…
def flatten(l):
return (item for sublist in l for item in sublist) # note the parens
Note, of course, that the sort of comprehension will only "flatten" a list of lists (or list of other iterables). Also if you pass it a list of strings you'll "flatten" it into a list of characters.
To generalize this in a meaningful way you first want to be able to cleanly distinguish between strings (or bytearrays) and other types of sequences (or other Iterables). So let's start with a simple function:
import collections
def non_str_seq(p):
'''p is putatively a sequence and not a string nor bytearray'''
return isinstance(p, collections.Iterable) and not (isinstance(p, str) or isinstance(p, bytearray))
Using that we can then build a recursive function to flatten any
def flatten(s):
'''Recursively flatten any sequence of objects
'''
results = list()
if non_str_seq(s):
for each in s:
results.extend(flatten(each))
else:
results.append(s)
return results
There are probably more elegant ways to do this. But this works for all the Python built-in types that I know of. Simple objects (numbers, strings, instances of None, True, False are all returned wrapped in list. Dictionaries are returned as lists of keys (in hash order).
My question seems simple, but for a novice to python like myself this is starting to get too complex for me to get, so here's the situation:
I need to take a list such as:
L = [(a, b, c), (d, e, d), (etc, etc, etc), (etc, etc, etc)]
and make each index an individual list so that I may pull elements from each index specifically. The problem is that the list I am actually working with contains hundreds of indices such as the ones above and I cannot make something like:
L_new = list(L['insert specific index here'])
for each one as that would mean filling up the memory with hundreds of lists corresponding to individual indices of the first list and would be far too time and memory consuming from my point of view. So my question is this, how can I separate those indices and then pull individual parts from them without needing to create hundreds of individual lists (at least to the point where I wont need hundreds of individual lines to create them).
I might be misreading your question, but I'm inclined to say that you don't actually have to do anything to be able to index your tuples. See my comment, but: L[0][0] will give "a", L[0][1] will give "b", L[2][1] will give "etc" etc...
If you really want a clean way to turn this into a list of lists you could use a list comprehension:
cast = [list(entry) for entry in L]
In response to your comment: if you want to access across dimensions I would suggest list comprehension. For your comment specifically:
crosscut = [entry[0] for entry in L]
In response to comment 2: This is largely a part of a really useful operation called slicing. Specifically to do the referenced operation you would do this:
multiple_index = [entry[0:3] for entry in L]
Depending on your readability preferences there are actually a number of possibilities here:
list_of_lists = []
for sublist in L:
list_of_lists.append(list(sublist))
iterator = iter(L)
for i in range(0,iterator.__length_hint__()):
return list(iterator.next())
# Or yield list(iterator.next()) if you want lazy evaluation
What you have there is a list of tuples, access them like a list of lists
L[3][2]
will get the second element from the 3rd tuple in your list L
Two way of using inner lists:
for index, sublist in enumerate(L):
# do something with sublist
pass
or with an iterator
iterator = iter(L)
sublist = L.next() # <-- yields the first sublist
in both case, sublist elements can be reached via
direct index
sublist[2]
iteration
iterator = iter(sublist)
iterator.next() # <-- yields first elem of sublist
for elem in sublist:
# do something with my elem
pass
I would like to extend a list while looping over it:
for idx in xrange(len(a_list)):
item = a_list[idx]
a_list.extend(fun(item))
(fun is a function that returns a list.)
Question:
Is this already the best way to do it, or is something nicer and more compact possible?
Remarks:
from matplotlib.cbook import flatten
a_list.extend(flatten(fun(item) for item in a_list))
should work but I do not want my code to depend on matplotlib.
for item in a_list:
a_list.extend(fun(item))
would be nice enough for my taste but seems to cause an infinite loop.
Context:
I have have a large number of nodes (in a dict) and some of them are special because they are on the boundary.
'a_list' contains the keys of these special/boundary nodes. Sometimes nodes are added and then every new node that is on the boundary needs to be added to 'a_list'. The new boundary nodes can be determined by the old boundary nodes (expresses here by 'fun') and every boundary node can add several new nodes.
Have you tried list comprehensions? This would work by creating a separate list in memory, then assigning it to your original list once the comprehension is complete. Basically its the same as your second example, but instead of importing a flattening function, it flattens it through stacked list comprehensions. [edit Matthias: changed + to +=]
a_list += [x for lst in [fun(item) for item in a_list] for x in lst]
EDIT: To explain what going on.
So the first thing that will happen is this part in the middle of the above code:
[fun(item) for item in a_list]
This will apply fun to every item in a_list and add it to a new list. Problem is, because fun(item) returns a list, now we have a list of lists. So we run a second (stacked) list comprehension to loop through all the lists in our new list that we just created in the original comprehension:
for lst in [fun(item) for item in a_list]
This will allow us to loop through all the lists in order. So then:
[x for lst in [fun(item) for item in a_list] for x in lst]
This means take every x (that is, every item) in every lst (all the lists we created in our original comprehension) and add it to a new list.
Hope this is clearer. If not, I'm always willing to elaborate further.
Using itertools, it can be written as:
import itertools
a_list += itertools.chain(* itertools.imap(fun, a_list))
or, if you're aiming for code golf:
a_list += sum(map(fun, a_list), [])
Alternatively, just write it out:
new_elements = map(fun, a_list) # itertools.imap in Python 2.x
for ne in new_elements:
a_list.extend(ne)
As you want to extend the list, but loop only over the original list, you can loop over a copy instead of the original:
for item in a_list[:]:
a_list.extend(fun(item))
Using generator
original_list = [1, 2]
original_list.extend((x for x in original_list[:]))
# [1, 2, 1, 2]
if I have a list, is there any way to check if it contains any other lists?
what i mean to say is, I want to know if a list has this strcuture: [] as opposed to this structure [[]]
so, compare [1,2,3,4] to [1,[2,3],4]
this is complicated by the fact that i have a list of strings.
well, phihag's solution seems to be working so far, but what I'm doing is this:
uniqueCrossTabs = list(itertools.chain.from_iterable(uniqueCrossTabs))
in order to flatten a list if it has other lists in it.
But since my list contains strings, if this is done on an already flattened list, I get a list of each character of each string that was in the original list.
This is not the behavior i was looking for. so, checking to see if the list needs to be flattened before flattening is neccessary.
any(isinstance(el, list) for el in input_list)
You can take phihag's answer even further if you actually want a list of all the lists inside the list:
output_list = filter( lambda x: isinstance(x,list), input_list)
lst1 in lst2
Yields True iff lst1 is in lst2.