I have a numpy array of hex string (eg: ['9', 'A', 'B']) and want to convert them all to integers between 0 255. The only way I know how to do this is use a for loop and append a seperate numpy array.
import numpy as np
hexArray = np.array(['9', 'A', 'B'])
intArray = np.array([])
for value in hexArray:
intArray = np.append(intArray, [int(value, 16)])
print(intArray) # output: [ 9. 10. 11.]
Is there a better way to do this?
A vectorized way with array's-view functionality -
In [65]: v = hexArray.view(np.uint8)[::4]
In [66]: np.where(v>64,v-55,v-48)
Out[66]: array([ 9, 10, 11], dtype=uint8)
Timings
Setup with given sample scaled-up by 1000x -
In [75]: hexArray = np.array(['9', 'A', 'B'])
In [76]: hexArray = np.tile(hexArray,1000)
# #tianlinhe's soln
In [77]: %timeit [int(value, 16) for value in hexArray]
1.08 ms ± 5.67 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# #FBruzzesi soln
In [78]: %timeit list(map(functools.partial(int, base=16), hexArray))
1.5 ms ± 40.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# From this post
In [79]: %%timeit
...: v = hexArray.view(np.uint8)[::4]
...: np.where(v>64,v-55,v-48)
15.9 µs ± 294 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
With the use of list comprehension:
array1=[int(value, 16) for value in hexArray]
print (array1)
output:
[9, 10, 11]
Alternative using map:
import functools
list(map(functools.partial(int, base=16), hexArray))
[9, 10, 11]
intArray = [int(hexNum, 16) for hexNum in list(hexArray)]
Try this, uses list comprehension to convert each hexadecimal number to an integer.
Here is another good one:
int_array = np.frompyfunc(int, 2, 1) #Can be used, for example, to add broadcasting to a built-in Python function
int_array(hexArray,16).astype(np.uint32)
If you want to know more about it: https://numpy.org/doc/stable/reference/generated/numpy.frompyfunc.html?highlight=frompyfunc#numpy.frompyfunc
Check out the speed:
import numpy as np
import functools
hexArray = np.array(['ffaa', 'aa91', 'b1f6'])
hexArray = np.tile(hexArray,1000)
def x_test(hexArray):
v = hexArray.view(np.uint32)[::4]
return np.where(v > 64, v - 55, v - 48)
int_array = np.frompyfunc(int, 2, 1)
%timeit -n 100 int_array(hexArray,16).astype(np.uint32)
%timeit -n 100 np.fromiter(map(functools.partial(int, base=16), hexArray),dtype=np.uint32)
%timeit -n 100 [int(value, 16) for value in hexArray]
%timeit -n 100 x_test(hexArray)
print(f'\n\n{int_array(hexArray,16).astype(np.uint32)=}\n{np.fromiter(map(functools.partial(int, base=16), hexArray),dtype=np.uint32)=}\n{[int(value, 16) for value in hexArray][:10]=}\n{x_test(hexArray)=}')
460 µs ± 2.42 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
1.25 ms ± 2.66 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
1.11 ms ± 6.56 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
16.8 µs ± 165 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
int_array(hexArray,16).astype(np.uint32)=array([65450, 43665, 45558, ..., 65450, 43665, 45558], dtype=uint32)
np.fromiter(map(functools.partial(int, base=16), hexArray),dtype=np.uint32)=array([65450, 43665, 45558, ..., 65450, 43665, 45558], dtype=uint32)
[int(value, 16) for value in hexArray][:10]=[65450, 43665, 45558, 65450, 43665, 45558, 65450, 43665, 45558, 65450]
x_test(hexArray)=array([47, 42, 43, ..., 47, 42, 43], dtype=uint32)
Divakar's answer is the fastest, but, unfortunately, does not work for bigger hex numbers (at least for me)
Related
I am unsure as to the cost of transforming a matrix of tuples into a list form which is easier to manipulate. The main priority is being able to change a column of the matrix as fast as possible
I have a matrix in the form of
[(a,b,c),(d,e,f),(g,h,i)]
which can appear in any size of n x m but for this example we'll take 3x3 matrix.
my main goal is to be able to change the values of any column in the matrix (only one at a time) (eg (b,e,h)).
my initial attempt was to transform the matrix into a list ie
[[a,b,c],[d,e,f],[g,h,i]]
which would be easier
but I feel that it would be costly in terms of transforming every tuple into a list and back into a tuple.
My main question could be how to optimize this to its fullest?
In [37]: def change_column_list_comp(old_m, col, value):
...: return [
...: tuple(list(row[:col]) + [value] + list(row[col + 1:]))
...: for row in old_m
...: ]
...:
In [38]: def change_column_list_convert(old_m, col, value):
...: list_m = list(map(list, old_m))
...: for row in list_m:
...: row[col] = value
...:
...: return list(map(tuple, list_m))
...:
In [39]: m = [tuple('abc'), tuple('def'), tuple('ghi')]
In [40]: %timeit change_column_list_comp(m, 1, 2)
2.05 µs ± 89.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [41]: %timeit change_column_list_convert(m, 1, 2)
1.28 µs ± 121 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Looks like converting to a list, modifying the values, and converting back to tuple is faster. Note that this may not be the most efficient way of writing these functions.
However, these functions seem to start to converge as we scale up our matrix.
In [6]: m_100k = [tuple(string.printable)] * 100_000
In [7]: %timeit change_column_list_comp(m_100k, 1, 2)
163 ms ± 3.7 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [8]: %timeit change_column_list_convert(m_100k, 1, 2)
117 ms ± 5.67 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [42]: m_1m = [tuple(string.printable)] * 1_000_000
In [43]: %timeit change_column_list_comp(m_1m, 1, 2)
1.72 s ± 74.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [44]: %timeit change_column_list_convert(m_1m, 1, 2)
1.24 s ± 84.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
At the end of the day you should be using the right tools for the job. While it's not really in the OP, it's just worth mentioning that numpy is simply the better way to go.
In [13]: m_np = np.array([list('abc'), list('def'), list('ghi')])
In [17]: %timeit m_np[:, 1] = 2; m_np
610 ns ± 48.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [20]: m_np_100k = np.array([[string.printable] * 100_000])
In [21]: %timeit m_np_100k[:, 1] = 2; m_np_100k
545 ns ± 63.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [22]: m_np_1m = np.array([[string.printable] * 1_000_000])
# This might be using cached data
In [23]: %timeit m_np_1m[:, 1] = 2; m_np_1m
515 ns ± 31.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
# Avoiding cache
In [24]: %timeit m_np_1m[:, 4] = 9; m_np_1m
557 ns ± 37.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
This might not be the fairest comparison as we're manually returning the matrix, but you can see there is significant improvement.
I have two tensors: tensor a and tensor b. How can I change some value of tensor a according to the value of tensor b?
I know the codes following are right, but it runs pretty slow when the tensor is big. Is there any other method?
import torch
a = torch.rand(10).cuda()
b = torch.rand(10).cuda()
a[b > 0.5] = 0.
For this exact use case also consider
a * (b <= 0.5)
which seems to be the fastest out of the following
In [1]: import torch
...: a = torch.rand(3**10)
...: b = torch.rand(3**10)
In [2]: %timeit a[b > 0.5] = 0.
553 µs ± 17.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [3]: a = torch.rand(3**10)
In [4]: %timeit temp = torch.where(b > 0.5, torch.tensor(0.), a)
...:
49 µs ± 391 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [5]: a = torch.rand(3**10)
In [6]: %timeit temp = (a * (b <= 0.5))
44 µs ± 381 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [7]: %timeit a.masked_fill_(b > 0.5, 0.)
244 µs ± 3.48 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
I guess torch.where would be faster I have measure in CPU here is result.
import torch
a = torch.rand(3**10)
b = torch.rand(3**10)
%timeit a[b > 0.5] = 0.
852 µs ± 30.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit temp = torch.where(b > 0.5, torch.tensor(0.), a)
294 µs ± 4.51 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
I'm trying to element-wise multiply two arrays to form a single string.
Can anyone advise?
import numpy as np
def array_translate(array):
intlist = [x for x in array if isinstance(x, int)]
strlist = [x for x in array if isinstance(x, str)]
joinedlist = np.multiply(intlist, strlist)
return "".join(joinedlist)
print(array_translate(["Cat", 2, "Dog", 3, "Mouse", 1])) # => "CatCatDogDogDogMouse"
I receive this error:
File "/Users/peteryoon/PycharmProjects/Test3/Test3.py", line 8, in array_translate
joinedlist = np.multiply(intlist, strlist)
numpy.core._exceptions.UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U21'), dtype('<U21')) -> dtype('<U21')
I was able to solve using list comprehension below. But curious to see how numpy works.
def array_translate(array):
intlist = [x for x in array if isinstance(x, int)]
strlist = [x for x in array if isinstance(x, str)]
return "".join(intlist*strlist for intlist, strlist in zip(intlist, strlist))
print(array_translate(["Cat", 2, "Dog", 3, "Mouse", 1])) # => "CatCatDogDogDogMouse"
In [79]: arr = np.array(['Cat','Dog','Mouse'])
In [80]: cnt = np.array([2,3,1])
Timings for various alternatives. The relative placement may vary with the size of the arrays (and whether you start with lists or arrays). So do your own testing:
In [93]: timeit ''.join(np.repeat(arr,cnt))
7.98 µs ± 57.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [94]: timeit ''.join([str(wd)*i for wd,i in zip(arr,cnt)])
5.96 µs ± 167 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [95]: timeit ''.join(arr.astype(object)*cnt)
13.3 µs ± 50.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [96]: timeit ''.join(np.char.multiply(arr,cnt))
27.4 µs ± 307 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [100]: timeit ''.join(np.frompyfunc(lambda w,i: w*i,2,1)(arr,cnt))
10.4 µs ± 164 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [101]: %%timeit f = np.frompyfunc(lambda w,i: w*i,2,1)
...: ''.join(f(arr,cnt))
7.95 µs ± 93.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [102]: %%timeit x=arr.tolist(); y=cnt.tolist()
...: ''.join([str(wd)*i for wd,i in zip(x,y)])
1.36 µs ± 39.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
np.repeat works for all kinds of arrays.
List comprehension uses the string multiply, and shouldn't be dismissed out of hand. Often it is fastest, especially if starting with lists.
Object dtype converts the string dtype to Python strings, and then delegates the action to the string multiply.
np.char applies string methods to elements of an array. While convenient, it seldom is fast.
edit
In [104]: timeit ''.join(np.repeat(arr,cnt).tolist())
4.04 µs ± 197 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
perhaps using repeat
z = array(['Cat', 'Dog', 'Mouse'], dtype='<U5')
"".join(np.repeat(z, (2, 3, 1)))
'CatCatDogDogDogMouse'
This question already has answers here:
How can I sum every n array values and place the result into a new array? [duplicate]
(3 answers)
Closed 3 years ago.
For example,
I have a numpy array containing:
[1, 2, 3, 4, 5, 6]
I want to create an array as follows:
[3, 7, 11]
That is, I want to add the two neighboring elements into a new one.
I have tried the obvious:
for i in range(0, predictions.shape[0]+1, 2):
new_pred = np.append(new_pred, (predictions[i] + predictions[i+1]) / 2)
print(predictions.shape)
(16000, 0)
print(new_pred.shape)
(87998, 0)
But the dimension of new_pred is not half of 16000.
So I am wondering is there anything wrong with my code? And is there a convenient way to implement it?
There are many different possibilities, here it is one, neither the slowest one nor the fastest, of them,
>>> import numpy as np
>>> a = np.arange(30)
>>> a.reshape(-1, 2).sum(axis=1)
array([ 1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41, 45, 49, 53, 57])
>>>
For the record (please note that we have a new fastest answer that, imho, can't be bettered at all)
In [17]: a = np.arange(10**5)
In [18]: %timeit a.reshape(-1,2).sum(axis=1)
1.08 ms ± 1.35 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [19]: %timeit [(a[i]+ a[i+1]) for i in range(0, len(a-1), 2)]
23.4 ms ± 117 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [20]: %timeit [sum(item) for ind, item in enumerate(zip(a, a[1:])) if ind%2 == 0]
49.9 ms ± 313 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [21]: %timeit [sum(item) for item in zip(a[::2], a[1::2])]
30.2 ms ± 91 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
...
In [23]: %timeit a[::2]+a[1::2]
78.9 µs ± 79.9 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Use slices of ndarray:
predictions[::2] + predictions[1::2]
It is 10 times faster than "reshape" solution
>>> a = np.arange(10**5)
>>> timeit(lambda: a.reshape(-1,2).sum(axis=-1), number=1000)
0.785971520585008
>>> timeit(lambda: a[::2]+a[1::2], number=1000)
0.07569492445327342
another pythonic Possibility would be to use list comprehensions:
something like this for the example you posted:
import numpy as np
a = np.arange(1, 7)
res = [(a[i]+ a[i+1]) for i in range(0, len(a-1), 2)]
print(res)
hope it helps
Using zip
zip_ls = zip(ls[::2], ls[1::2])
new_ls = [sum(item) for item in zip_ls]
I have a pandas series with boolean entries. I would like to get a list of indices where the values are True.
For example the input pd.Series([True, False, True, True, False, False, False, True])
should yield the output [0,2,3,7].
I can do it with a list comprehension, but is there something cleaner or faster?
Using Boolean Indexing
>>> s = pd.Series([True, False, True, True, False, False, False, True])
>>> s[s].index
Int64Index([0, 2, 3, 7], dtype='int64')
If need a np.array object, get the .values
>>> s[s].index.values
array([0, 2, 3, 7])
Using np.nonzero
>>> np.nonzero(s)
(array([0, 2, 3, 7]),)
Using np.flatnonzero
>>> np.flatnonzero(s)
array([0, 2, 3, 7])
Using np.where
>>> np.where(s)[0]
array([0, 2, 3, 7])
Using np.argwhere
>>> np.argwhere(s).ravel()
array([0, 2, 3, 7])
Using pd.Series.index
>>> s.index[s]
array([0, 2, 3, 7])
Using python's built-in filter
>>> [*filter(s.get, s.index)]
[0, 2, 3, 7]
Using list comprehension
>>> [i for i in s.index if s[i]]
[0, 2, 3, 7]
As an addition to rafaelc's answer, here are the according times (from quickest to slowest) for the following setup
import numpy as np
import pandas as pd
s = pd.Series([x > 0.5 for x in np.random.random(size=1000)])
Using np.where
>>> timeit np.where(s)[0]
12.7 µs ± 77.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Using np.flatnonzero
>>> timeit np.flatnonzero(s)
18 µs ± 508 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Using pd.Series.index
The time difference to boolean indexing was really surprising to me, since the boolean indexing is usually more used.
>>> timeit s.index[s]
82.2 µs ± 38.9 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Using Boolean Indexing
>>> timeit s[s].index
1.75 ms ± 2.16 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
If you need a np.array object, get the .values
>>> timeit s[s].index.values
1.76 ms ± 3.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
If you need a slightly easier to read version <-- not in original answer
>>> timeit s[s==True].index
1.89 ms ± 3.52 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Using pd.Series.where <-- not in original answer
>>> timeit s.where(s).dropna().index
2.22 ms ± 3.32 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> timeit s.where(s == True).dropna().index
2.37 ms ± 2.19 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Using pd.Series.mask <-- not in original answer
>>> timeit s.mask(s).dropna().index
2.29 ms ± 1.43 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> timeit s.mask(s == True).dropna().index
2.44 ms ± 5.82 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Using list comprehension
>>> timeit [i for i in s.index if s[i]]
13.7 ms ± 40.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Using python's built-in filter
>>> timeit [*filter(s.get, s.index)]
14.2 ms ± 28.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Using np.nonzero <-- did not work out of the box for me
>>> timeit np.nonzero(s)
ValueError: Length of passed values is 1, index implies 1000.
Using np.argwhere <-- did not work out of the box for me
>>> timeit np.argwhere(s).ravel()
ValueError: Length of passed values is 1, index implies 1000.
Also works:
s.where(lambda x: x).dropna().index, and
it has the advantage of being easy to chain pipe - if your series is being computed on the fly, you don't need to assign it to a variable.
Note that if s is computed from r: s = cond(r)
than you can also use: r.where(lambda x: cond(x)).dropna().index.
You can use pipe or loc to chain the operation, this is helpful when s is an intermediate result and you don't want to name it.
s = pd.Series([True, False, True, True, False, False, False, True], index=list('ABCDEFGH'))
out = s.pipe(lambda s_: s_[s_].index)
# or
out = s.pipe(lambda s_: s_[s_]).index
# or
out = s.loc[lambda s_: s_].index
print(out)
Index(['A', 'C', 'D', 'H'], dtype='object')