Develop Reference

Most Efficient Way To Separate Out Data from List within a List

I have a function with outputs 2 arrays, lets call them X_i, and Y_i which are two N x 1 arrays, where N is the number of points. By using multiprocessing's pool.apply_aync, I was able to parallelize this function which gave me my results in a HUGE list. The structure of the results are a list of M values, where each value is a list containing X_i and Y_i. So in summary, I have a huge list which M smaller lists containing the two arrays X_i and Y_i.
Now I have want append all the X_i's into one array called X and Y_i's called Y. What is the most efficient way to do this? I'm looking for some sort of parallel algorithm. Order does NOT matter!
So far I have just a simple for loop that separates this massive data array:
X = np.zeros((N,1))
Y = np.zeros((N,1))
for i in range(len(results))
X = np.append(results[i][0].reshape(N,1),X,axis = 1)
Y = np.append(results[i][1].reshape(N,1),Y,axis = 1)
I found this algorthim to be rather slow, so I need to speed it up! Thanks!

You should provide a simple scenario of your problem, break it down and give us a simple input, output scenario, it would help a lot, as all this variables and text make it a bit confusing.Maybe this can help;
You can unpack the lists, then grab the ones you need by index, append the list to your new empty X[] and append the other list you needed to Y[], at the end get the arrays out of the lists and merge those into your new N dimensional array or into a new list.
list = [[[1,2],[3,4]],[[4,5],[6,7]]]
sub_pre = []
flat_list = []
for sublist in list:
sub_pre.append(sublist)
for item in sublist:
flat_list.append(item)
print(list)
print(flat_list)

Thanks to #JonSG for the brilliant insight. This type of sorting algorithm can be sped up using array manipulation. Through the use of most parallels packages, a function that outputs in multiple arrays will most likely get put into a huge list. Here I have a list called results, which contains M smaller lists of two N x 1 arrays.
To unpack the main array and sort all the X_i and Y_i into their own X and Y arrays respectively, it can be done so like this.
np.shape(results) = (M, 2, N)
X = np.array(results)[:,0,:]
Y = np.array(results)[:,1,:]
This gave me an 100x speed increase!

Add n elements numpy array

I have a numpy array and I want to add n elements with the same value until the length of the array reaches 100.
For example
my_array = numpy.array([3, 4, 5])
Note that I do not know the length of the array beforehand. It may be anything 3 <= x <= 100
I want to add (100 - x) more elements, all with the value 9.
How can I do it?

There are two ways to approach this: concatenating arrays or assigning them.
You can use np.concatenate and generate an appropriately sized array:
my_array = # as you defined it
remainder = [9] * (100 - len(my_array))
remainder = np.array(remainder)
a100 = np.concatenate((my_array, remainder))
Alternatively, you can construct a np.full array, and then overwrite some of the values using slice notation:
a100 = numpy.full(100, 9)
my_array = # as you defined it
a100[0:len(my_array)] = my_array

It's important to remember with numpy arrays, you can't add elements like you can with lists. So adding numbers to an array is not really the best thing to do.
Far better is to start with an array, and replace the elements with new data as it comes in. For example:
import numpy as np
MY_SPECIAL_NUMBER = 100
my_array = np.array[3, 4, 5]
my_new_array = np.ones(100) * MY_SPECIAL_NUMBER
my_new_array[:my_array.size] = my_array
my_new_array is now what you want.

If you "cannot" know the size of your mysterious array:
fillvalue=9
padding=numpy.ones(100)*fillvalue
newarray=numpy.append(myarray, padding)
newarray=newarray[:100]

Efficient way for generating N arrays of random numbers between different ranges

I want to generate N arrays of fixed length n of random numbers with numpy, but arrays must have numbers varying between different ranges.
So for example, I want to generate N=100 arrays of size n=5 and each array must have its numbers between:
First number between 0 and 10
Second number between 20 and 100
and so on...
First idea that comes to my mind is doing something like:
first=np.random.randint(0,11, 100)
second=np.random.randint(20,101, 100)
...
And then I should nest them, Is there a more efficient way?

I would just put them inside another array and iterate them through their index
from np.random import randint
array_holder = [[] for i in range(N)] # Get N arrays in a holder
ab_holder = [[a1, b1], [a2, b2]]
for i in range(len(array_holder)): # You iterate over each array
[a, b] = [ab_holder[i][0], ab_holder[i][1]]
for j in range(size): # Size is the ammount of elements you want in each array
array_holder[i].append(randint(a, b)) # Where a is your base and b ends the range

Another possibility. Setting ranges indicates both what the ranges of the individual parts of each arrays must be and how many there are. size is the number of values to sample in each individual part of an array. N is the size of the Monte-Carlo sample. arrays is the result.
import numpy as np
ranges = [ (0, 10), (20, 100) ]
size = 5
N = 100
arrays = [ ]
for n in range(N):
one_array = []
for r in ranges:
chunk = np.random.randint(*r, size=size)
one_array.append(chunk)
arrays.append(one_array)
It might make an appreciable difference to use numpy's append in place of Python's but I've written this in this way to make it easier to read (and to write :)).

Single Line Nested For Loops in Python

I'm having some trouble understanding how nested single line for loops work. Here's an example:
I have this code:
NewArray = np.array([ get_position(i, size-1, t) for i in range(0, size)])
and I'm trying to rewrite this to:
for i in range(0,size):
NewArray = np.array([ get_position(i, size-1, t)])
But I'm getting different outputs, so I'm guessing there's a logic error here.
Could you point out the issue?
Thank you

It's because the first one creates a numpy array containing all your values (you create all values because you're using a list comprehension) and the second one creates a new array containing the last value each iteration (and it discards the array created in the last iteration because you reuse the same name).
You could rewrite the second one as:
lst = []
for i in range(0,size):
lst.append(get_position(i, size-1, t))
NewArray = np.array(lst)
that should give the same result as your first operation.

In the first you create an array of length size.
In the second you repeatedly (size times) create an array of length 1.

Python matrix indexing

I have the following code
l = len(time) #time is a 300 element list
ll = len(sample) #sample has 3 sublists each with 300 elements
w, h = ll, l
Matrix = [[0 for x in range(w)] for y in range(h)]
for n in range(0,l):
for m in range(0,ll):
x=sample[m]
Matrix[m][n]= x
When I run the code to fill the matrix I get an error message saying "list index out of range" I put in a print statement to see where the error happens and when m=0 and n=3 the matrix goes out of index.
from what I understand on the fourth line of the code I initialize a 3X300 matrix so why does it go out of index at 0X3 ?

You need to change Matrix[m][n]= x to Matrix[n][m]= x

The indexing of nested lists happens from the outside in. So for your code, you'll probably want:
Matrix[n][m] = x
If you prefer the other order, you can build the matrix differently (swap w and h in the list comprehensions).
Note that if you're going to be doing mathematical operations with this matrix, you may want to be using numpy arrays instead of Python lists. They're almost certainly going to be much more efficient at doing math operations than anything you can write yourself in pure Python.

Note that indexing in nested lists in Python happens from outside in, and so you'll have to change the order in which you index into your array, as follows:
Matrix[n][m] = x
For mathematical operations and matrix manipulations, using numpy two-dimensional arrays, is almost always a better choice. You can read more about them here.