Develop Reference

forcing a creation of 1d numpy array from a list/array of possibly iterable objects

I'm trying to see if there is a prettier way to create (i.e force the creation) of a 1d numpy array from another list/array of objects. These objects, however, may have entries that are themselves iterable (so they can be lists, tuples, etc. but can also be more arbitrary objects).
So to make things really simple, let me consider the following scenario:
a=[(1,2), (3,4), (3,5)]
b=np.array(a, dtype=object)
b.shape # gives (2,3), but I would like to have (3,1) or (3,)
I was wondering if there is a nice pythonic/numpy'ish way to force b to have a shape (3,), and the iterable structure of the elements of a to be neglected in b. Right now I do this:
a=[(1,2), (3,4), (3,5)]
b=np.empty(len(a), dtype=object)
for i,x in enumerate(a):
b[i]=x
b.shape # gives (3,) this is what i want.
which works, but a bit ugly. I could not find a nicer way to do this in way that's more built-in into numpy. Any ideas?
(more context: what I really need to do is reshuffle the dimensions of b in various ways, hence I don't want b to know anything about the dimensions of its elements if they are iterable).
Thanks!

In [60]: b = np.empty(3, object)
You don't need to iterate when assigning from a list:
In [61]: b[:] = [(1,2),(3,4),(3,5)]
In [62]: b
Out[62]: array([(1, 2), (3, 4), (3, 5)], dtype=object)
In [63]: b.shape
Out[63]: (3,)
For an array it doesn't work:
In [64]: b[:] = np.array([(1,2),(3,4),(3,5)])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-64-3042dce1f885> in <module>
----> 1 b[:] = np.array([(1,2),(3,4),(3,5)])
ValueError: could not broadcast input array from shape (3,2) into shape (3)
You may have use the iteration in the array case:
In [66]: for i,n in enumerate(np.array([(1,2),(3,4),(3,5)])):
...: b[i] = n
...:
In [67]: b
Out[67]: array([array([1, 2]), array([3, 4]), array([3, 5])], dtype=object)
Keep in mind that object dtype arrays are a bit of fall back option. np.array(...) tries to create a multidimensional array if possible (with numeric dtype). Making an object dtype is done only if that isn't possible. And for some combinations of shapes, it throws up its hands and raises an error.
Turning that array into a list of arrays with list() also works (same speed):
In [92]: b[:] = list(np.array([(1,2),(3,4),(3,5)]))
In [93]: b
Out[93]: array([array([1, 2]), array([3, 4]), array([3, 5])], dtype=object)

Error (only size-1 arrays can be converted to Python scalars) when trying Calculating maximus

Iam trying to calculate maximas from the below code but getting error as
TypeError: only size-1 arrays can be converted to Python scalars
Input_list= [1,9,96,9,7,4,3,77,0,2,3,4,5].
Please help
import ast,sys
import numpy as np
input_str = sys.stdin.read()
input_list = ast.literal_eval(input_str)
from scipy.signal import argrelextrema
Ar=np.array(input_list)
maximas=argrelextrema(Ar, np.greater) #store your final list here
maximas=[int(x) for x in maximas] #do not change this code, the output should be an integer list for evaluation purposes
print(maximas)

argrelextrema returns a tuple containing the arrays (https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.argrelextrema.html). For your case, you probably want to access the first element of this tuple, so something like
maximas=[int(x) for x in maximas[0]]

According to the docs, the scypy.signal.argrelaxtrema function returns a tuple of arrays, with one value for each dimension of the input. If the input is one-dimensional, you'll get a 1-tuple back.
I suspect you can fix your code with maximas = argrelextrema(Ar, np.greater)[0], or maybe maximas, = argrelextrema(Ar, np.greater) (note the comma after the variable name, which makes it an unpacking!).

In [295]: from scipy.signal import argrelextrema
In [296]: Ar = np.array([1,9,96,9,7,4,3,77,0,2,3,4,5])
In [297]: maximas=argrelextrema(Ar, np.greater)
When you ask about an error, you should show all of the error message, or at least enough so both you and we know were it occurs. That's useful information.
In [300]: [int(x) for x in maximas]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-300-0250728a7b76> in <module>
----> 1 [int(x) for x in maximas]
<ipython-input-300-0250728a7b76> in <listcomp>(.0)
----> 1 [int(x) for x in maximas]
TypeError: only size-1 arrays can be converted to Python scalars
Most likely the error is in the int() call, since int is meant to return one integer, either from a string or a numeric input. The error indicates that x was an array with more than one value.
Look at maximas:
In [301]: maximas
Out[301]: (array([2, 7]),)
It's a tuple, containing one array. So iteration on that tuple will produce an array.
We can select the array from the tuple with indexing:
In [302]: maximas[0]
Out[302]: array([2, 7])
In [303]: maximas[0].tolist() # convert it to a list
Out[303]: [2, 7]
In [304]: maximas[0].astype(int).tolist() # ensure it's integers
Out[304]: [2, 7]
According to the docs the result of argrelextrema is a tuple, one array per dimension of the input. In fact the code shows it returning np.nonzero(results). More commonly we use the np.where(results) alias.
There was no need to use int(x). maximas arrays are integer, indices.
You might not need to convert maximas to a list. The result of np.where can often be used directly in indexing, for example:
In [307]: Ar
Out[307]: array([ 1, 9, 96, 9, 7, 4, 3, 77, 0, 2, 3, 4, 5])
In [308]: Ar[maximas]
Out[308]: array([96, 77])
===
Be ware that if the input array is 2d, maximas will be a 2 element tuple.
In [312]: Be = Ar[:12].reshape(6,2) # 2d array
In [313]: argrelextrema(Be, np.greater)
Out[313]: (array([1, 3]), array([0, 1])) # 2 element tuple
In [314]: Be[_] # using the tuple to index Be
Out[314]: array([96, 77])

Mapping an integer to array (Python): ValueError: setting an array element with a sequence

I have a defaultdict which maps certain integers to a numpy array of size 20.
In addition, I have an existing array of indices. I want to turn that array of indices into a 2D array, where each original index is converted into an array via my defaultdict.
Finally, in the case that an index isn't found in the defaultdict, I want to create an array of zeros for that index.
Here's what I have so far
converter = lambda x: np.zeros((d), dtype='float32') if x == -1 else cVf[x]
vfunc = np.vectorize(converter)
cvf = vfunc(indices)
np.zeros((d), dtype='float32') and cVf[x] are identical data types/ shapes:
(Pdb) np.shape(cVf[0])
(20,)
Yet I get the error in the title (*** ValueError: setting an array element with a sequence.) when I try to run this code.
Any ideas?

You should give us a some sample arrays or dictionaries (in the case of cVF, so we can make a test run.
Read what vectorize has to say about the return value. Since you don't define otypes, it makes a test calculation to determine the dtype of the returned array. My first thought was that the test calc and subsequent one might be returning different things. But you claim converter will always be returning the same dtype and shape array.
But let's try something simpler:
In [609]: fv = np.vectorize(lambda x: np.array([x,x]))
In [610]: fv([1,2,3])
...
ValueError: setting an array element with a sequence.
It's having trouble with returning any array.
But if I give an otypes, it works
In [611]: fv = np.vectorize(lambda x: np.array([x,x]), otypes=[object])
In [612]: fv([1,2,3])
Out[612]: array([array([1, 1]), array([2, 2]), array([3, 3])], dtype=object)
In fact in this case I could use frompyfunc, which returns object dtype, and is the underlying function for vectorize (and a bit faster).
In [613]: fv = np.frompyfunc(lambda x: np.array([x,x]), 1,1)
In [614]: fv([1,2,3])
Out[614]: array([array([1, 1]), array([2, 2]), array([3, 3])], dtype=object)
vectorize and frompyfunc are designed for functions that are scalar in- scalar out. That scalar may be an object, even array, but is still treated as a scalar.

Numpy: apply along axis returns error: Setting an array element with a sequence

a very quick and simple error that I can't figure out to save my life:
temp = np.array([[5,0,3,5,6,0],
[2,2,1,3,0,0],
[5,3,4,5,3,4]])
def myfunc(x):
return x[np.nonzero(x)]
np.apply_along_axis(myfunc, axis=1, arr=temp)
Expected output is the non-zero numbers of each ROW of my temp array:
[5,3,5,6],[2,2,1,3],[5,3,4,5,3,4]
However, I'm getting the error:ValueError: setting an array element with a sequence.
If I simply do it without apply_along_axis, it works:
# prints [5,3,5,6]
print temp[0][np.nonzero(temp[0])]
The weird thing is that, if I just add a np.mean() to the myfunc return to the first code block above, it works as expected:
# This works as expected
temp = np.array([[5,0,3,5,6,0],
[2,2,1,3,0,0],
[5,3,4,5,3,4]])
def myfunc(x):
return np.mean(x[np.nonzero(x)])
np.apply_along_axis(myfunc, axis=1, arr=temp)
I suspect it's something to do with how apply_along_axis work underneath the hood. Any tips will be appreciated!

As mentioned in the documentation -
Returns: apply_along_axis : ndarray The output array. The shape of
outarr is identical to the shape of arr, except along the axis
dimension, where the length of outarr is equal to the size of the
return value of func1d. If func1d returns a scalar outarr will have
one fewer dimensions than arr.
Because of the inconsistent shapes of the output at different iterations, it seems we are getting that error.
Now, to solve your problem, let me suggest a method with np.nonzero on the entire array and then splitting the second output from it -
In [165]: temp = np.array([[5,0,3,5,6,0],
...: [2,2,1,3,0,0],
...: [5,3,4,5,3,4]])
In [166]: r,c = np.nonzero(temp)
...: idx = np.unique(r,return_index=1)[1]
...: out = np.split(c,idx[1:])
...:
In [167]: out
Out[167]: [array([0, 2, 3, 4]), array([0, 1, 2, 3]), array([0, 1, 2, 3, 4, 5])]

In numpy 1.13, this definition should work:
def myfunc(x):
res = np.empty((), dtype=object)
res[()] = x[np.nonzero(x)]
return res
By returning a 0d array containing the array, numpy will not try and stack the subarrays.

how can I flatten an 2d numpy array, which has different length in the second axis?

I have a numpy array which looks like:
myArray = np.array([[1,2],[3]])
But I can not flatten it,
In: myArray.flatten()
Out: array([[1, 2], [3]], dtype=object)
If I change the array to the same length in the second axis, then I can flatten it.
In: myArray2 = np.array([[1,2],[3,4]])
In: myArray2.flatten()
Out: array([1, 2, 3, 4])
My Question is:
Can I use some thing like myArray.flatten() regardless the dimension of the array and the length of its elements, and get the output: array([1,2,3])?

myArray is a 1-dimensional array of objects. Your list objects will simply remain in the same order with flatten() or ravel(). You can use hstack to stack the arrays in sequence horizontally:
>>> np.hstack(myArray)
array([1, 2, 3])
Note that this is basically equivalent to using concatenate with an axis of 1 (this should make sense intuitively):
>>> np.concatenate(myArray, axis=1)
array([1, 2, 3])
If you don't have this issue however and can merge the items, it is always preferable to use flatten() or ravel() for performance:
In [1]: u = timeit.Timer('np.hstack(np.array([[1,2],[3,4]]))'\
....: , setup = 'import numpy as np')
In [2]: print u.timeit()
11.0124390125
In [3]: u = timeit.Timer('np.array([[1,2],[3,4]]).flatten()'\
....: , setup = 'import numpy as np')
In [4]: print u.timeit()
3.05757689476
Iluengo's answer also has you covered for further information as to why you cannot use flatten() or ravel() given your array type.

Well, I agree with the other answers when they say that hstack or concatenate do the job in this case. However, I would like to point that even if it 'fixes' the problem, the problem is not addressed properly.
The problem is that even if it looks like the second axis has different length, this is not true in practice. If you try:
>>> myArray.shape
(2,)
>>> myArray.dtype
dtype('O') # stands for Object
>>> myArray[0]
[1, 2]
It shows you that your array is not a 2D array with variable size (as you might think), it is just a 1D array of objects. In your case, the elements are list, being the first element of your array a 2-element list and the second element of the array is a 1-element list.
So, flatten and ravel won't work because transforming 1D array to a 1D array results in exactly the same 1D array. If you have a object numpy array, it won't care about what you put inside, it will treat individual items as unkown items and can't decide how to merge them.
What you should have in consideration, is if this is the behaviour you want for your application. Numpy arrays are specially efficient with fixed-size numeric matrices. If you are playing with arrays of objects, I don't see why would you like to use Numpy instead of regular python lists.

np.hstack works in this case
In [69]: np.hstack(myArray)
Out[69]: array([1, 2, 3])