Develop Reference

How work to with an array of arrays and how initialize multiple arrays in Numpy?

I have to write an ABM (agent-based model) project in Python and I need to initialize 50 agents which will each contain a different set of numbers. I cannot use a matrix of 50 rows because each agent (each row) can have a different quantity of elements so the vector of each agent has not the same length: when certain conditions for agent_i occur in the algorithm then a number calculated by the algorithm is added to its vector.
The simplest way would be to write manually every one like this
agent_1 = np.array([])
agent_2 = np.array([])
agent_3 = np.array([])
...
but of course I can't. I don't know if exist a way to automatically initialize with a loop, something like
for i in range(50):
agent_i = np.array([])
If it exist, It would be useful because then, when certain conditions occur in the algorithm, I could add a calculated numbers to agent_i:
agent_i = np.append(agent_i, result_of_algorithm)
Maybe another way is to use an array of arrays
[[agent_1_collection],[agent_2_collection], ... , [agent_50_collection]]
Once again, I don't know how to initialize an array of arrays and I don't know how to add a number to a specific array: in fact I think it can't be done like this (assume, for simplicity, that I have this little array of only 3 agents and that I know how it is done):
vect = np.array([[1],[2,3],[4,5,6]])
result_of_algorithm_for_agent_2 = ...some calculations that we assume give as result... = 90
vect[1] = np.append(vect[1], result_of_algorithm_for_agent_2)
output:
array([[1], array([ 2, 3, 90]), [4, 5, 6]], dtype=object)
why it changes in that way?
Do you have any advice on how to manipulate arrays of arrays? For example, how to add elements to a specific point of a sub-array (agent)?
Thanks.

List of Arrays
You can create a list of arrays:
agents = [np.array([]) for _ in range(50)]
And then to append values to some agent, say agents[0], use:
items_to_append = [1, 2, 3] # Or whatever you want.
agents[0] = np.append(agents[0], items_to_append)
List of Lists
Alternatively, if you don't need to use np.arrays, you can use lists for the agents values. In that case you would initialize with:
a = [[] for _ in range(50)]
And you can append to agents[0] with either
single_value = 1 # Or whatever you want.
agents[0].append(single_value)
Or with
items_to_append = [1, 2, 3] # Or whatever you want
agents[0] += items_to_append

Python: Elegant way to combine output variables of a function run many times

I have a function that returns many output arrays of varying size.
arr1,arr2,arr3,arr4,arr5, ... = func(data)
I want to run this function many times over a time series of data, and combine each output variable into one array that covers the whole time series.
To elaborate: If the output arr1 has dimensions (x,y) when the function is called, I want to run the function 't' times and end up with an array that has dimensions (x,y,t). A list of 't' arrays with size (x,y) would also be acceptable, but not preferred.
Again, the output arrays do not all have the same dimensions, or even the same number of dimensions. Arr2 might have size (x2,y2), arr3 might be only a vector of length (x3). I do not know the size of all of these arrays before hand.
My current solution is something like this:
arr1 = []
arr2 = []
arr3 = []
...
for t in range(t_max):
arr1_t, arr2_t, arr3_t, ... = func(data[t])
arr1.append(arr1_t)
arr2.append(arr2_t)
arr3.append(arr3_t)
...
and so on. However this is inelegant looking when repeated 27 times for each output array.
Is there a better way to do this?

You can just make arr1, arr2, etc. a list of lists (of vectors or matrices or whatever). Then use a loop to iterate the results obtained from func and add them to the individual lists.
arrN = [[] for _ in range(N)] # N being number of results from func
for t in range(t_max):
results = func(data[t])
for i, res in enumerate(results):
arrN[i].append(res)
The elements in the different sub-lists do not have to have the same dimensions.

Not sure if it counts as "elegant", but you can build a list of the result tuples then use zip to group them into tuples by return position instead of by call number, then optionally map to convert those tuples to the final data type. For example, with numpy array:
from future_builtins import map, zip # Only on Python 2, to minimize temporaries
import numpy as np
def func(x):
'Dumb function to return tuple of powers of x from 1 to 27'
return tuple(x ** i for i in range(1, 28))
# Example inputs for func
data = [np.array([[x]*10]*10, dtype=np.uint8) for in range(10)]
# Output is generator of results for each call to func
outputs = map(func, data)
# Pass each complete result of func as a positional argument to zip via star
# unpacking to regroup, so the first return from each func call is the first
# group, then the second return the second group, etc.
positional_groups = zip(*outputs)
# Convert regrouped data (`tuple`s of 2D results) to numpy 3D result type, unpack to names
arr1,arr2,arr3,arr4,arr5, ...,arr27 = map(np.array, positional_groups)
If the elements returned from func at a given position might have inconsistent dimensions (e.g. one call might return 10x10 as the first return, and another 5x5), you'd avoid the final map step (since the array wouldn't have consistent dimensions and just replace the second-to last step with:
arr1,arr2,arr3,arr4,arr5, ...,arr27 = zip(*outputs)
making arr# a tuple of 2D arrays, or if the need to be mutable:
arr1,arr2,arr3,arr4,arr5, ...,arr27 = map(list, zip(*outputs))
to make them lists of 2D arrays.

This answer gives a solution using structured arrays. It has the following requirement: Ggven a function f that returns N arrays, and the size of each of the returned arrays can be different -- then for all results of f, len(array_i) must always be same. eg.
arrs_a = f("a")
arrs_b = f("b")
for sub_arr_a, sub_arr_b in zip(arrs_a, arrs_b):
assert len(sub_arr_a) == len(sub_arr_b)
If the above is true, then you can use structured arrays. A structured array is like a normal array, just with a complex data type. For instance, I could specify a data type that is made up of one array of ints of shape 5, and a second array of floats of shape (2, 2). eg.
# define what a record looks like
dtype = [
# tuples of (field_name, data_type)
("a", "5i4"), # array of five 4-byte ints
("b", "(2,2)f8"), # 2x2 array of 8-byte floats
]
Using dtype you can create a structured array, and set all the results on the structured array in one go.
import numpy as np
def func(n):
"mock implementation of func"
return (
np.ones(5) * n,
np.ones((2,2))* n
)
# define what a record looks like
dtype = [
# tuples of (field_name, data_type)
("a", "5i4"), # array of five 4-byte ints
("b", "(2,2)f8"), # 2x2 array of 8-byte floats
]
size = 5
# create array
arr = np.empty(size, dtype=dtype)
# fill in values
for i in range(size):
# func must return a tuple
# or you must convert the returned value to a tuple
arr[i] = func(i)
# alternate way of instantiating arr
arr = np.fromiter((func(i) for i in range(size)), dtype=dtype, count=size)
# How to use structured arrays
# access individual record
print(arr[1]) # prints ([1, 1, 1, 1, 1], [[1, 1], [1, 1]])
# access specific value -- get second record -> get b field -> get value at 0,0
assert arr[2]['b'][0,0] == 2
# access all values of a specific field
print(arr['a']) # prints all the a arrays

using python itertools to manage nested for loops

I am trying to use itertools.product to manage the bookkeeping of some nested for loops, where the number of nested loops is not known in advance. Below is a specific example where I have chosen two nested for loops; the choice of two is only for clarity, what I need is a solution that works for an arbitrary number of loops.
This question provides an extension/generalization of the question appearing here:
Efficient algorithm for evaluating a 1-d array of functions on a same-length 1d numpy array
Now I am extending the above technique using an itertools trick I learned here:
Iterating over an unknown number of nested loops in python
Preamble:
from itertools import product
def trivial_functional(i, j): return lambda x : (i+j)*x
idx1 = [1, 2, 3, 4]
idx2 = [5, 6, 7]
joint = [idx1, idx2]
func_table = []
for items in product(*joint):
f = trivial_functional(*items)
func_table.append(f)
At the end of the above itertools loop, I have a 12-element, 1-d array of functions, func_table, each element having been built from the trivial_functional.
Question:
Suppose I am given a pair of integers, (i_1, i_2), where these integers are to be interpreted as the indices of idx1 and idx2, respectively. How can I use itertools.product to determine the correct corresponding element of the func_table array?
I know how to hack the answer by writing my own function that mimics the itertools.product bookkeeping, but surely there is a built-in feature of itertools.product that is intended for exactly this purpose?

I don't know of a way of calculating the flat index other than doing it yourself. Fortunately this isn't that difficult:
def product_flat_index(factors, indices):
if len(factors) == 1: return indices[0]
else: return indices[0] * len(factors[0]) + product_flat_index(factors[1:], indices[1:])
>> product_flat_index(joint, (2, 1))
9
An alternative approach is to store the results in a nested array in the first place, making translation unnecessary, though this is more complex:
from functools import reduce
from operator import getitem, setitem, itemgetter
def get_items(container, indices):
return reduce(getitem, indices, container)
def set_items(container, indices, value):
c = reduce(getitem, indices[:-1], container)
setitem(c, indices[-1], value)
def initialize_table(lengths):
if len(lengths) == 1: return [0] * lengths[0]
subtable = initialize_table(lengths[1:])
return [subtable[:] for _ in range(lengths[0])]
func_table = initialize_table(list(map(len, joint)))
for items in product(*map(enumerate, joint)):
f = trivial_functional(*map(itemgetter(1), items))
set_items(func_table, list(map(itemgetter(0), items)), f)
>>> get_items(func_table, (2, 1)) # same as func_table[2][1]
<function>

So numerous answers were quite useful, thanks to everyone for the solutions.
It turns out that if I recast the problem slightly with Numpy, I can accomplish the same bookkeeping, and solve the problem I was trying to solve with vastly improved speed relative to pure python solutions. The trick is just to use Numpy's reshape method together with the normal multi-dimensional array indexing syntax.
Here's how this works. We just convert func_table into a Numpy array, and reshape it:
func_table = np.array(func_table)
component_dimensions = [len(idx1), len(idx2)]
func_table = np.array(func_table).reshape(component_dimensions)
Now func_table can be used to return the correct function not just for a single 2d point, but for a full array of 2d points:
dim1_pts = [3,1,2,1,3,3,1,3,0]
dim2_pts = [0,1,2,1,2,0,1,2,1]
func_array = func_table[dim1_pts, dim2_pts]
As usual, Numpy to the rescue!

This is a little messy, but here you go:
from itertools import product
def trivial_functional(i, j): return lambda x : (i+j)*x
idx1 = [1, 2, 3, 4]
idx2 = [5, 6, 7]
joint = [enumerate(idx1), enumerate(idx2)]
func_map = {}
for indexes, items in map(lambda x: zip(*x), product(*joint)):
f = trivial_functional(*items)
func_map[indexes] = f
print(func_map[(2, 0)](5)) # 40 = (3+5)*5

I'd suggest using enumerate() in the right place:
from itertools import product
def trivial_functional(i, j): return lambda x : (i+j)*x
idx1 = [1, 2, 3, 4]
idx2 = [5, 6, 7]
joint = [idx1, idx2]
func_table = []
for items in product(*joint):
f = trivial_functional(*items)
func_table.append(f)
From what I understood from your comments and your code, func_table is simply indexed by the occurence of a certain input in the sequence. You can access it back again using:
for index, items in enumerate(product(*joint)):
# because of the append(), index is now the
# position of the function created from the
# respective tuple in join()
func_table[index](some_value)

Two dimensional array in python

I want to know how to declare a two dimensional array in Python.
arr = [[]]
arr[0].append("aa1")
arr[0].append("aa2")
arr[1].append("bb1")
arr[1].append("bb2")
arr[1].append("bb3")
The first two assignments work fine. But when I try to do, arr[1].append("bb1"), I get the following error:
IndexError: list index out of range.
Am I doing anything silly in trying to declare the 2-D array?
Edit:
but I do not know the number of elements in the array (both rows and columns).

You do not "declare" arrays or anything else in python. You simply assign to a (new) variable. If you want a multidimensional array, simply add a new array as an array element.
arr = []
arr.append([])
arr[0].append('aa1')
arr[0].append('aa2')
or
arr = []
arr.append(['aa1', 'aa2'])

There aren't multidimensional arrays as such in Python, what you have is a list containing other lists.
>>> arr = [[]]
>>> len(arr)
1
What you have done is declare a list containing a single list. So arr[0] contains a list but arr[1] is not defined.
You can define a list containing two lists as follows:
arr = [[],[]]
Or to define a longer list you could use:
>>> arr = [[] for _ in range(5)]
>>> arr
[[], [], [], [], []]
What you shouldn't do is this:
arr = [[]] * 3
As this puts the same list in all three places in the container list:
>>> arr[0].append('test')
>>> arr
[['test'], ['test'], ['test']]

What you're using here are not arrays, but lists (of lists).
If you want multidimensional arrays in Python, you can use Numpy arrays. You'd need to know the shape in advance.
For example:
import numpy as np
arr = np.empty((3, 2), dtype=object)
arr[0, 1] = 'abc'

You try to append to second element in array, but it does not exist.
Create it.
arr = [[]]
arr[0].append("aa1")
arr[0].append("aa2")
arr.append([])
arr[1].append("bb1")
arr[1].append("bb2")
arr[1].append("bb3")

We can create multidimensional array dynamically as follows,
Create 2 variables to read x and y from standard input:
print("Enter the value of x: ")
x=int(input())
print("Enter the value of y: ")
y=int(input())
Create an array of list with initial values filled with 0 or anything using the following code
z=[[0 for row in range(0,x)] for col in range(0,y)]
creates number of rows and columns for your array data.
Read data from standard input:
for i in range(x):
for j in range(y):
z[i][j]=input()
Display the Result:
for i in range(x):
for j in range(y):
print(z[i][j],end=' ')
print("\n")
or use another way to display above dynamically created array is,
for row in z:
print(row)

When constructing multi-dimensional lists in Python I usually use something similar to ThiefMaster's solution, but rather than appending items to index 0, then appending items to index 1, etc., I always use index -1 which is automatically the index of the last item in the array.
i.e.
arr = []
arr.append([])
arr[-1].append("aa1")
arr[-1].append("aa2")
arr.append([])
arr[-1].append("bb1")
arr[-1].append("bb2")
arr[-1].append("bb3")
will produce the 2D-array (actually a list of lists) you're after.

You can first append elements to the initialized array and then for convenience, you can convert it into a numpy array.
import numpy as np
a = [] # declare null array
a.append(['aa1']) # append elements
a.append(['aa2'])
a.append(['aa3'])
print(a)
a_np = np.asarray(a) # convert to numpy array
print(a_np)

a = [[] for index in range(1, n)]

For compititve programming
1) For input the value in an 2D-Array
row=input()
main_list=[]
for i in range(0,row):
temp_list=map(int,raw_input().split(" "))
main_list.append(temp_list)
2) For displaying 2D Array
for i in range(0,row):
for j in range(0,len(main_list[0]):
print main_list[i][j],
print

the above method did not work for me for a for loop, where I wanted to transfer data from a 2D array to a new array under an if the condition. This method would work
a_2d_list = [[1, 2], [3, 4]]
a_2d_list.append([5, 6])
print(a_2d_list)
OUTPUT - [[1, 2], [3, 4], [5, 6]]

x=3#rows
y=3#columns
a=[]#create an empty list first
for i in range(x):
a.append([0]*y)#And again append empty lists to original list
for j in range(y):
a[i][j]=input("Enter the value")

In my case I had to do this:
for index, user in enumerate(users):
table_body.append([])
table_body[index].append(user.user.id)
table_body[index].append(user.user.username)
Output:
[[1, 'john'], [2, 'bill']]

How to create nested lists in python?

I know you can create easily nested lists in python like this:
[[1,2],[3,4]]
But how to create a 3x3x3 matrix of zeroes?
[[[0] * 3 for i in range(0, 3)] for j in range (0,3)]
or
[[[0]*3]*3]*3
Doesn't seem right. There is no way to create it just passing a list of dimensions to a method? Ex:
CreateArray([3,3,3])

In case a matrix is actually what you are looking for, consider the numpy package.
http://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html#numpy.zeros
This will give you a 3x3x3 array of zeros:
numpy.zeros((3,3,3))
You also benefit from the convenience features of a module built for scientific computing.

List comprehensions are just syntactic sugar for adding expressiveness to list initialization; in your case, I would not use them at all, and go for a simple nested loop.
On a completely different level: do you think the n-dimensional array of NumPy could be a better approach?
Although you can use lists to implement multi-dimensional matrices, I think they are not the best tool for that goal.

NumPy addresses this problem
Link
>>> a = array( [2,3,4] )
>>> a
array([2, 3, 4])
>>> type(a)
<type 'numpy.ndarray'>
But if you want to use the Python native lists as a matrix the following helper methods can become handy:
import copy
def Create(dimensions, item):
for dimension in dimensions:
item = map(copy.copy, [item] * dimension)
return item
def Get(matrix, position):
for index in position:
matrix = matrix[index]
return matrix
def Set(matrix, position, value):
for index in position[:-1]:
matrix = matrix[index]
matrix[position[-1]] = value

Or use the nest function defined here, combined with repeat(0) from the itertools module:
nest(itertools.repeat(0),[3,3,3])

Just nest the multiplication syntax:
[[[0] * 3] * 3] * 3
It's therefore simple to express this operation using folds
def zeros(dimensions):
return reduce(lambda x, d: [x] * d, [0] + dimensions)
Or if you want to avoid reference replication, so altering one item won't affect any other you should instead use copies:
import copy
def zeros(dimensions):
item = 0
for dimension in dimensions:
item = map(copy.copy, [item] * dimension)
return item