Develop Reference

Error in converting an array to 2D matrix in Python

I have the problem regarding the output of this algorithm. For example: for input chunk([1, 2, 3, 4, 5, 6, 7, 8], 3) it should return [[ 1, 2, 3], [4, 5, 6], [7, 8, '']] but instead it returns [[7, 8, 6], [7, 8, 6], [7, 8, 6]].
However, when m_list is defined under the loop for r in range(rows):, it returns correct value.
I can't figure out why it returns wrong value if m_list is defined outside the loop for r in range(rows):. What could be the reason ?
# --- Directions
# Given an array and chunk size, divide the array into many subarrays
# where each subarray is of length size
# --- Examples
# chunk([1, 2, 3, 4], 2) --> [[ 1, 2], [3, 4]]
# chunk([1, 2, 3, 4, 5], 2) --> [[ 1, 2], [3, 4], [5, '']]
# chunk([1, 2, 3, 4, 5, 6, 7], 3) --> [[ 1, 2, 3], [4, 5, 6], [7, '', '']]
import math
def chunk (array, size):
rows = 0
l = len(array)
if l % size == 0:
rows = l/size
else:
rows = int(math.floor(l/size) + 1)
m_list = ['' for e in range(size)]
m_matrix = [['' for g in range(size)] for w in range(rows)]
i = 0
for r in range(rows):
for u in range(size):
if i == l:
break
else:
m_list[u] = array[i]
i += 1
m_matrix[r] = m_list
return m_matrix
length = int(raw_input('how many elements you want in the array?: '))
m_inputArray = ['' for q in range(length)]
print 'Debug0:--> ' + str(m_inputArray)
for z in range(length):
p = int(raw_input('Enter the value at index %i: ' %(z)))
m_inputArray[z] = p
m_inputSize = int(raw_input('Enter the size: '))
result = chunk(m_inputArray, m_inputSize)
print result

There are several things wrong with your code. Firstly every loop of u the start value of m_list is the previous list (so the first time it is ['','',''] but the second time it is [1,2,3], and the third time it is [4,5,6]. Which means that since the third time only one value is left in the array, only the first value in the m_list gets redefined, resulting in an m_list of [7,5,6].
Secondly, by saying: m_matrix[u] = m_list you are creating a reference to m_list, you are not copying m_list into m_matrix. This means that once m_list changes, so do the values in m_matrix. Which means in the end you will have defined m_matrix to be [m_list,m_list,m_list], resulting in your results of [[7,5,6],[7,5,6],[7,5,6]]. A solution for this would be to make slice of m_list, like this: m_matrix = m_list[:].
This is how I would do the whole thing:
def chunk(inputarray,size):
array = inputarray[:]
m_matrix = []
while len(array) > 0:
if len(array[:size]) < size:
array.extend(['' for j in range(size-len(array[:size]))])
m_matrix.append(array[:size])
del array[:size]
return m_matrix
If you don't need the original array anymore you can also remove the array = inputarray[:] line of code. Also, probably not the fastest/best way of doing this, but I just wanted to provide something quick. This was done in python 2.7, so if you're using another version you might have to alter some things.

seems a bit over complicated. this is what i came up with.
written for python 3 but does work in 2.
def pop_with_replace(array, index=0, blank=''):
try:
return array.pop(index)
except IndexError:
return blank
def chunk(array, size):
out = []
while array:
t_list = []
for i in range(size):
t_list.append(pop_with_replace(array))
out.append(t_list)
return out
if __name__ == '__main__':
print(chunk(list(range(10)), 3))
there's some things we could change as well. like removing this method pop_with_replace for a ternary operator? i didn't put this in the first solution as they can be awkward to read if not used to them.
t_list.append(array.pop() if array else '')
looking at this we could roll it all up into a list comp. but we're starting to get hard to read.
While array:
out.append([array.pop(0) if array else '' for x in range(size)]
but it does leave the final code looking nice and small.
def chunk(array, size):
out = []
while array:
out.append([array.pop(0) if array else '' for x in range(size)])
return out

In this example there is only one m_list where you update values and final result is [m_list, m_list, m_list ... m_list].
If you define m_list in loop new list will be created in each loop pass.
You can assign directly to m_matrix[r][u] = array[i]
Note that you are with list of lists, not true matrix, and m_matrix[r] = m_list replaces list on r-th position with reference to m_list list.

Making list of number with integers and moving them Python 2.7 [duplicate]

This question already has answers here:
Efficient way to rotate a list in python
(27 answers)
Closed 6 years ago.
I'm trying to make integers containing within value to shift from leftmost to rightmost. For example if values is [1,2,3,4,5] after running the code it would be [2,3,4,5,1].
I managed to write the code but I some how made the value shift from rightmost to leftmost which is other way around. Like [5,1,2,3,4].
Been sitting for hours trying to figure it out, any hands? Thank you
values = [1,2,3,4,5]
temp = values[len(values) - 1]
for index in range(len(values) - 1, 0, -1):
values[index] = values[index - 1]
values[0] = temp
print values

This kind of rotation is most easily implemented using slice concatenation:
def rotate(lst, n):
n = n % len(lst) # allows rotations beyond length
return lst[n:] + lst[:n]
> rotate([1,2,3,4,5], 1) # rotate forward
[2, 3, 4, 5, 1]
> rotate([1,2,3,4,5], -2) # rotate backward
[4, 5, 1, 2, 3]
Repairing your original code, however:
temp = values[0] # store the first value cause it'll be overwritten in loop
for index in range(len(values) - 1):
values[index] = values[index + 1] # shift values to the left
values[len(values) - 1] = temp # write former first to last

In your case, knowing the length of the list is not necessary.
You can just use list slicing:
values[1:] + values[:1]
Example:
In [8]: values = [1,2,3,4,5]
In [9]: values[1:] + values[:1]
Out[9]: [2, 3, 4, 5, 1]

You could cycle through the list using mutability, like this:
def cycle(input_list):
list_copy = input_list[:]
list_copy.append(list_copy.pop(0))
return list_copy
# Output
>>> input_ist = [i for i in range(1, 6)] # Same as [1,2,3,4,5]
>>> cycle(input_list)
[2, 3, 4, 5, 1]

Printing tree levels in reverse

I have a code where it prints the Huffman Tree. It is this part:
while len(numArr) > 1:
numArr = [numArr[0] + numArr[1]] + numArr[2:]
numArr = sorted(numArr)
valHold = numArr[0] * 8
print(numArr)
Don't mind the valHold variables I use it to compute for the uncompressed bits of the input string.
Let's say I have 1,1,1,2,3,4 as the elements of list numArr (the elements comes from a Counter and transferred to letter_arr and numArr to separate the two).
I can only print it like this:
1,1,1,1,2,3,4
1,1,2,2,3,4
2,2,2,3,4
2,3,4,4
4,4,5
5,8
13
Is there a way I can print it the other way? The way it will more look like a tree? Like this:
13
5,8
4,4,5
2,3,4,4
2,2,2,3,4
1,1,2,2,3,4
1,1,1,1,2,3,4
It will be much better if you can teach me how to print it with indent:
13
5,8
4,4,5
2,3,4,4
2,2,2,3,4
1,1,2,2,3,4
1,1,1,1,2,3,4
Please note that the elements of the numArr list is not predefined. It is based on what the user inputs in the program.

Sure:
tree = []
while len(numArr) > 1:
numArr = [numArr[0] + numArr[1]] + numArr[2:]
numArr = sorted(numArr)
valHold = numArr[0] * 8
tree.append(numArr)
indent = len(tree)
for row in tree[::-1]:
print(" " * indent, row)
indent -= 1

You could output your data in a tree format as follows:
numArray = [
[1, 2, 1, 4, 1, 1, 3],
[2, 4, 1, 3, 2, 1],
[2, 3, 2, 4, 2],
[4, 2, 3, 4],
[5, 4, 4],
[8, 5],
[13]]
output = [','.join(str(x) for x in sorted(row)) for row in numArray[::-1]]
for row in output:
print row.center(len(output[-1]))
This would display:
13
5,8
4,4,5
2,3,4,4
2,2,2,3,4
1,1,2,2,3,4
1,1,1,1,2,3,4
[::-1] can be used to read an array in reverse order. So the idea here is to read each row and convert each element into a string. These are then joined using commas to create a list of numbers. Finally each row is then displayed centred based on the length of the longest entry.

In order to print in reverse order, you can put it in a list first and later reverse it.
array = []
while len(numArr) > 1:
numArr = [numArr[0] + numArr[1]] + numArr[2:]
numArr = sorted(numArr)
array.append(numArr)
array.reverse()
To indent the output and align it with each number might need a little extra work, but you can try to center the output as a work-around.
First convert each list to a string, and compute maximum width. Then use str.center to center the text.
array_str = list(map(lambda level: ','.join(str(i) for i in level), array))
width = max(len(s) for s in array_str)
for s in array_str:
print(s.center(width))

Python: compute average of n-th elements in list of lists with different lengths

Suppose I have the following list of lists:
a = [
[1, 2, 3],
[2, 3, 4],
[3, 4, 5, 6]
]
I want to have the average of each n-th element in the arrays. However, when wanting to do this in a simple way, Python generated out-of-bounds errors because of the different lengths. I solved this by giving each array the length of the longest array, and filling the missing values with None.
Unfortunately, doing this made it impossible to compute an average, so I converted the arrays into masked arrays. The code shown below works, but it seems rather cumbersome.
import numpy as np
import numpy.ma as ma
a = [ [1, 2, 3],
[2, 3, 4],
[3, 4, 5, 6] ]
# Determine the length of the longest list
lenlist = []
for i in a:
lenlist.append(len(i))
max = np.amax(lenlist)
# Fill each list up with None's until required length is reached
for i in a:
if len(i) <= max:
for j in range(max - len(i)):
i.append(None)
# Fill temp_array up with the n-th element
# and add it to temp_array
temp_list = []
masked_arrays = []
for j in range(max):
for i in range(len(a)):
temp_list.append(a[i][j])
masked_arrays.append(ma.masked_values(temp_list, None))
del temp_list[:]
# Compute the average of each array
avg_array = []
for i in masked_arrays:
avg_array.append(np.ma.average(i))
print avg_array
Is there a way to do this more quickly? The final list of lists will contain 600000 'rows' and up to 100 'columns', so efficiency is quite important :-).

tertools.izip_longest would do all the padding with None's for you so your code can be reduced to:
import numpy as np
import numpy.ma as ma
from itertools import izip_longest
a = [ [1, 2, 3],
[2, 3, 4],
[3, 4, 5, 6] ]
averages = [np.ma.average(ma.masked_values(temp_list, None)) for temp_list in izip_longest(*a)]
print(averages)
[2.0, 3.0, 4.0, 6.0]
No idea what the fastest way in regard to the numpy logic but this is definitely going to be a lot more efficient than your own code.
If you wanted a faster pure python solution:
from itertools import izip_longest, imap
a = [[1, 2, 3],
[2, 3, 4],
[3, 4, 5, 6]]
def avg(x):
x = filter(None, x)
return sum(x, 0.0) / len(x)
filt = imap(avg, izip_longest(*a))
print(list(filt))
[2.0, 3.0, 4.0, 6.0]
If you have 0's in the arrays that won't work as 0 will be treated as Falsey, you will have to use a list comp to filter in that case but it will still be faster:
def avg(x):
x = [i for i in x if i is not None]
return sum(x, 0.0) / len(x)
filt = imap(avg, izip_longest(*a))

Here's an almost* fully vectorized solution based on np.bincount and np.cumsum -
# Store lengths of each list and their cumulative and entire summations
lens = np.array([len(i) for i in a]) # Only loop to get lengths
C = lens.cumsum()
N = lens.sum()
# Create ID array such that the first element of each list is 0,
# the second element as 1 and so on. This is needed in such a format
# for use with bincount later on.
shifts_arr = np.ones(N,dtype=int)
shifts_arr[C[:-1]] = -lens[:-1]+1
id_arr = shifts_arr.cumsum()-1
# Use bincount to get the summations and thus the
# averages across all lists based on their positions.
avg_out = np.bincount(id_arr,np.concatenate(a))/np.bincount(id_arr)
-* Almost because we are getting the lengths of lists with a loop, but with minimal computation involved there, must not affect the total runtime hugely.
Sample run -
In [109]: a = [ [1, 2, 3],
...: [2, 3, 4],
...: [3, 4, 5, 6] ]
In [110]: lens = np.array([len(i) for i in a])
...: C = lens.cumsum()
...: N = lens.sum()
...:
...: shifts_arr = np.ones(N,dtype=int)
...: shifts_arr[C[:-1]] = -lens[:-1]+1
...: id_arr = shifts_arr.cumsum()-1
...:
...: avg_out = np.bincount(id_arr,np.concatenate(a))/np.bincount(id_arr)
...:
In [111]: avg_out
Out[111]: array([ 2., 3., 4., 6.])

You can already clean your code to compute the max length: this single line does the job:
len(max(a,key=len))
Combining with other answer you will get the result like so:
[np.mean([x[i] for x in a if len(x) > i]) for i in range(len(max(a,key=len)))]

You can also avoid the masked array and use np.nan instead:
def replaceNoneTypes(x):
return tuple(np.nan if isinstance(y, type(None)) else y for y in x)
a = [np.nanmean(replaceNoneTypes(temp_list)) for temp_list in zip_longest(*df[column], fillvalue=np.nan)]

On your test array:
[np.mean([x[i] for x in a if len(x) > i]) for i in range(4)]
returns
[2.0, 3.0, 4.0, 6.0]

If you are using Python version >= 3.4, then import the statistics module
from statistics import mean
if using lower versions, create a function to calculate mean
def mean(array):
sum = 0
if (not(type(array) == list)):
print("there is some bad format in your input")
else:
for elements in array:
try:
sum = sum + float(elements)
except:
print("non numerical entry found")
average = (sum + 0.0) / len(array)
return average
Create a list of lists, for example
myList = [[1,2,3],[4,5,6,7,8],[9,10],[11,12,13,14],[15,16,17,18,19,20,21,22],[23]]
iterate through myList
for i, lists in enumerate(myList):
print(i, mean(lists))
This will print down the sequence n, and the average of nth list.
To find particularly the average of only nth list, create a function
def mean_nth(array, n):
if((type(n) == int) and n >= 1 and type(array) == list):
return mean(myList[n-1])
else:
print("there is some bad format of your input")
Note that index starts from zero, so for instance if you are looking for the mean of 5th list, it will be at index 4. this explains n-1 in the code.
And then call the function, for example
avg_5thList = mean_nth(myList, 5)
print(avg_5thList)
Running the above code on myList yields following result:
0 2.0
1 6.0
2 9.5
3 12.5
4 18.5
5 23.0
18.5
where the first six lines are generated from the iterative loop, and display the index of nth list and list average. Last line (18.5) displays the average of 5th list as a result of mean_nth(myList, 5) call.
Further, for a list like yours,
a = [
[1, 2, 3],
[2, 3, 4],
[3, 4, 5, 6]
]
Lets say you want average of 1st elements, i.e. (1+2+3)/3 = 2, or 2nd elements, i.e., (2+3+4)/3 = 3, or 4th elements such as 6/1 = 6, you will need to find the length of each list so that you can identify in the nth element exists in a list or not. For that, you first need to arrange your list of lists in the order of length of lists.
You can either
1) first sort the main list according to size of constituent lists iteratively, and then go through the sorted list to identify if the constituent lists are of sufficient length
2) or you can iteratively look into the original list for length of constituent lists.
(I can definitely get back with working out a faster recursive algorithm if needed)
Computationally second one is more efficient, so assuming that your 5th element means 4th in the index(0, 1, 2, 3, 4), or nth element means (n-1)th element, lets go with that and create a function
def find_nth_average(array, n):
if(not(type(n) == int and (int(n) >= 1))):
return "Bad input format for n"
else:
if (not(type(array) == list)):
return "Bad input format for main list"
else:
total = 0
count = 0
for i, elements in enumerate(array):
if(not(type(elements) == list)):
return("non list constituent found at location " + str(i+1))
else:
listLen = len(elements)
if(int(listLen) >= n):
try:
total = total + elements[n-1]
count = count + 1
except:
return ("non numerical entity found in constituent list " + str(i+1))
if(int(count) == 0):
return "No such n-element exists"
else:
average = float(total)/float(count)
return average
Now lets call this function on your list a
print(find_nth_average(a, 0))
print(find_nth_average(a, 1))
print(find_nth_average(a, 2))
print(find_nth_average(a, 3))
print(find_nth_average(a, 4))
print(find_nth_average(a, 5))
print(find_nth_average(a, 'q'))
print(find_nth_average(a, 2.3))
print(find_nth_average(5, 5))
The corresponding results are:
Bad input format for n
2.0
3.0
4.0
6.0
No such n-element exists
Bad input format for n
Bad input format for n
Bad input format for main list
If you have an erratic list, like
a = [[1, 2, 3], 2, [3, 4, 5, 6]]
that contains a non - list element, you get an output:
non list constituent found at location 2
If your constituent list is erratic, like:
a = [[1, 'p', 3], [2, 3, 4], [3, 4, 5, 6]]
that contains a non - numerical entity in a list, and find the average of 2nd elements by print(find_nth_average(a, 2))
you get an output:
non numerical entity found in constituent list 1

Extract elements of list at odd positions

So I want to create a list which is a sublist of some existing list.
For example,
L = [1, 2, 3, 4, 5, 6, 7], I want to create a sublist li such that li contains all the elements in L at odd positions.
While I can do it by
L = [1, 2, 3, 4, 5, 6, 7]
li = []
count = 0
for i in L:
if count % 2 == 1:
li.append(i)
count += 1
But I want to know if there is another way to do the same efficiently and in fewer number of steps.

Solution
Yes, you can:
l = L[1::2]
And this is all. The result will contain the elements placed on the following positions (0-based, so first element is at position 0, second at 1 etc.):
1, 3, 5
so the result (actual numbers) will be:
2, 4, 6
Explanation
The [1::2] at the end is just a notation for list slicing. Usually it is in the following form:
some_list[start:stop:step]
If we omitted start, the default (0) would be used. So the first element (at position 0, because the indexes are 0-based) would be selected. In this case the second element will be selected.
Because the second element is omitted, the default is being used (the end of the list). So the list is being iterated from the second element to the end.
We also provided third argument (step) which is 2. Which means that one element will be selected, the next will be skipped, and so on...
So, to sum up, in this case [1::2] means:
take the second element (which, by the way, is an odd element, if you judge from the index),
skip one element (because we have step=2, so we are skipping one, as a contrary to step=1 which is default),
take the next element,
Repeat steps 2.-3. until the end of the list is reached,
EDIT: #PreetKukreti gave a link for another explanation on Python's list slicing notation. See here: Explain Python's slice notation
Extras - replacing counter with enumerate()
In your code, you explicitly create and increase the counter. In Python this is not necessary, as you can enumerate through some iterable using enumerate():
for count, i in enumerate(L):
if count % 2 == 1:
l.append(i)
The above serves exactly the same purpose as the code you were using:
count = 0
for i in L:
if count % 2 == 1:
l.append(i)
count += 1
More on emulating for loops with counter in Python: Accessing the index in Python 'for' loops

For the odd positions, you probably want:
>>>> list_ = list(range(10))
>>>> print list_[1::2]
[1, 3, 5, 7, 9]
>>>>

I like List comprehensions because of their Math (Set) syntax. So how about this:
L = [1, 2, 3, 4, 5, 6, 7]
odd_numbers = [y for x,y in enumerate(L) if x%2 != 0]
even_numbers = [y for x,y in enumerate(L) if x%2 == 0]
Basically, if you enumerate over a list, you'll get the index x and the value y. What I'm doing here is putting the value y into the output list (even or odd) and using the index x to find out if that point is odd (x%2 != 0).

You can also use itertools.islice if you don't need to create a list but just want to iterate over the odd/even elements
import itertools
L = [1, 2, 3, 4, 5, 6, 7]
li = itertools.islice(l, 1, len(L), 2)

You can make use of bitwise AND operator &:
>>> x = [1, 2, 3, 4, 5, 6, 7]
>>> y = [i for i in x if i&1]
[1, 3, 5, 7]
This will give you the odd elements in the list. Now to extract the elements at odd indices you just need to change the above a bit:
>>> x = [10, 20, 30, 40, 50, 60, 70]
>>> y = [j for i, j in enumerate(x) if i&1]
[20, 40, 60]
Explanation
Bitwise AND operator is used with 1, and the reason it works is because, odd number when written in binary must have its first digit as 1. Let's check:
23 = 1 * (2**4) + 0 * (2**3) + 1 * (2**2) + 1 * (2**1) + 1 * (2**0) = 10111
14 = 1 * (2**3) + 1 * (2**2) + 1 * (2**1) + 0 * (2**0) = 1110
AND operation with 1 will only return 1 (1 in binary will also have last digit 1), iff the value is odd.
Check the Python Bitwise Operator page for more.
P.S: You can tactically use this method if you want to select odd and even columns in a dataframe. Let's say x and y coordinates of facial key-points are given as columns x1, y1, x2, etc... To normalize the x and y coordinates with width and height values of each image you can simply perform:
for i in range(df.shape[1]):
if i&1:
df.iloc[:, i] /= heights
else:
df.iloc[:, i] /= widths
This is not exactly related to the question but for data scientists and computer vision engineers this method could be useful.