Okay, so I've got a piece of Python code which really needs optimizing.
It's a Game-of-Life iteration over a small (80x60-pixel) image and extracts the RGB values from it.
currently using nested for-loops; I'd rather swap out those for loops for the faster map() c function, but if I do that I can't figure out how I can get the x,y values, nor the local values defined out of the scope of the functions I'd need to define.
would using map() be any faster than this current set of for loops? How could I use it and still get x,y?
I currently use pygame Surfaces, and I've tried the surfarray/pixelarray modules, but since I'm changing/getting every pixel, it's a lot slower than Surface.get_at()/set_at().
Also, slightly irrelevant... do you think this could be made quicker if Python wasn't traversing a list of numbers but just incrementing a number, like in other languages? Why doesn't python include a normal for() as well as their foreach()?
The amount of conditionals there probably makes things slower too, right? The slowest part is checking for neighbours (where it builds the list n)... I replaced that whole bit with slice access on a 2D array but it doesn't work properly.
Redacted version of code:
xr = xrange(80)
yr = xrange(60)
# surface is an instance of pygame.Surface
get_at = surface.get_at()
set_at = surface.set_at()
for x in xr:
# ....
for y in yr:
# ...
pixelR = get_at((x,y))[0]
pixelG = get_at((x,y))[1]
pixelB = get_at((x,y))[2]
# ... more complex stuff here which changes R,G,B values independently of each other
set_at((x,y),(pixelR,pixelG,pixelB))
Full version of the function:
# xr, yr = xrange(80), xrange(60)
def live(surface,xr,yr):
randint = random.randint
set_at = surface.set_at
get_at = surface.get_at
perfect = perfectNeighbours #
minN = minNeighbours # All global variables that're defined in a config file.
maxN = maxNeighbours #
pos = actual # actual = (80,60)
n = []
append = n.append
NEIGHBOURS = 0
for y in yr: # going height-first for aesthetic reasons.
decay = randint(1,maxDecay)
growth = randint(1,maxGrowth)
for x in xr:
r, g, b, a = get_at((x,y))
del n[:]
NEIGHBOURS = 0
if x>0 and y>0 and x<pos[0]-1 and y<pos[1]-1:
append(get_at((x-1,y-1))[1])
append(get_at((x+1,y-1))[1])
append(get_at((x,y-1))[1])
append(get_at((x-1,y))[1])
append(get_at((x+1,y))[1])
append(get_at((x-1,y+1))[1])
append(get_at((x+1,y+1))[1])
append(get_at((x,y+1))[1])
for a in n:
if a > 63:
NEIGHBOURS += 1
if NEIGHBOURS == 0 and (r,g,b) == (0,0,0): pass
else:
if NEIGHBOURS < minN or NEIGHBOURS > maxN:
g = 0
b = 0
elif NEIGHBOURS==perfect:
g += growth
if g > 255:
g = 255
b += growth
if b > growth: b = growth
else:
if g > 10: r = g-10
if g > 200: b = g-100
if r > growth: g = r
g -= decay
if g < 0:
g = 0
b = 0
r -= 1
if r < 0:
r = 0
set_at((x,y),(r,g,b))
What's making your code slow is probably not the loops, they are incredibly fast.
What slows done your code are the number of function calls. For example
pixelR = get_at((x,y))[0]
pixelG = get_at((x,y))[1]
pixelB = get_at((x,y))[2]
is a lot slower than (about 3 times I guess)
r, g, b, a = get_at((x,y))
Every get_at, set_at call locks the surface, therefore it's faster to directly access the pixels using the available methods. The one that seems most reasonable is Surface.get_buffer.
Using map doesn't work in your example, because you need the indexes. With as few as 80 and 60 numbers it might even be faster to use range() instead of xrange().
map(do_stuff, ((x, y) for x in xrange(80) for y in xrange(60)))
where do_stuff would presumably be defined like so:
def do_stuff(coords):
r, g, b, a = get_at(coords)
# ... whatever you need to do with those ...
set_at(coords, (r, g, b))
You could alternatively use a list comprehension instead of a generator expression as the second argument to map (replace ((x, y) ...) with [(x, y) ...]) and use range instead of xrange. I'd say that it's not very likely to have a significant effect on performance, though.
Edit: Note that gs is certainly right about the for loops not being the main thing in need of optimisation in your code... Cutting down on superfluous calls to get_at is more important. In fact, I'm not sure if replacing the loops with map will actually improve performance here at all... Having said that, I find the map version more readable (perhaps because of my FP background...), so here you go anyway. ;-)
Since you are reading and rewriting every pixel, I think you can get the best speed improvement by not using a Surface.
I suggest first taking your 80x60 image and converting it to a plain bitmap file with 32-bit pixels. Then read the pixel data into a python array object. Now you can walk over the array object, reading values, calculating new values, and poking the new values into place with maximum speed. When done, save your new bitmap image, and then convert it to a Surface.
You could also use 24-bit pixels, but that should be slower. 32-bit pixels means one pixel is one 32-bit integer value, which makes the array of pixels much easier to index. 24-bit packed pixels means each pixel is 3 bytes, which is much more annoying to index into.
I believe you will gain much more speed out of this approach than by trying to avoid the use of for. If you try this, please post something here to let us know how well it worked or didn't. Good luck.
EDIT: I thought that an array has only a single index. I'm not sure how you managed to get two indexes to work. I was expecting you to do something like this:
def __i(x, y):
assert(0 <= x < 80)
assert(0 <= y < 60)
i = (y*80 + x) * 4
return i
def red(x, y):
return __a[__i(x, y)]
def green(x, y):
return __a[__i(x, y) + 1]
def blue(x, y):
return __a[__i(x, y) + 2]
def rgb(x, y):
i = __i(x, y)
return __a[i], __a[i + 1], __a[i + 2]
def set_rgb(x, y, r, g, b):
i = __i(x, y)
_a[i] = r
_a[i + 1] = g
_a[i + 2] = b
# example:
r, g, b = rgb(23, 33)
Since a Python array can only hold a single type, you will want to set the type to "unsigned byte" and then index like I showed.
Where of course __a is the actual array variable.
If none of this is helpful, try converting your bitmap into a list, or perhaps three lists. You can use nested lists to get 2D addressing.
I hope this helps. If it is not helpful, then I am not understanding what you are doing; if you explain more I'll try to improve the answer.
Related
I'm trying to make an rgb color picture editor, using just numpy.
I've tried using a nested for loop, but it's really slow (over a minute).
I'm wanting to control first, second, and third element (r,g,b) of the third dimension of the nested array. Thanks
This is to just look at the numbers:
%matplotlib inline
import numpy as np
img = plt.imread('galaxy.jpg')
img = np.array(img)
for i in range(len(img)):
for j in range(len(img[i])):
for k in (img[i][j]):
print(k)
Perhaps this might help you. np.ndenumerate() lets you iterate through a matrix without nested for loops. I did a quick test and my second for loop (in the example below) is slightly faster than your triple nested for loop, as far as printing is concerned. Printing is very slow so taking out the print statements might help with speed. As far as modifying these values, I added r g b a variables that can be modified to scale the various pixel values. Just a thought, but perhaps it might give you more ideas to expand on. Also, I didn't check to see which index values correspond to r, g, b, or a.
r = 1.0
g = 1.0
b = 1.0
a = 1.0
for index, pixel in np.ndenumerate(img): # <--- Acheives the same as your original code
print(pixel)
for index, pixel in np.ndenumerate(img):
i = index[0]
j = index[1]
print("{} {} {} {}".format(img[i][j][0], img[i][j][1], img[i][j][2], img[i][j][3]))
for index, pixel in np.ndenumerate(img):
i = index[0]
j = index[1]
imgp[i][j][0] *= r;
imgp[i][j][1] *= g;
imgp[i][j][2] *= b;
imgp[i][j][3] *= a;
Hope this helps
I am basically looking for a faster/better/efficient way to perform a piece of my python code.
Here goes a simpler version of my part of code.
import numpy as np
A = np.random.choice(100,80) # randomly select integers
A = np.sort(A) # sort it
B = np.unique(A) # drop the duplicate values
What I want to do with this vector B is to remove its elements that fall within a given range from the previous value. For example, if I have a sorted vector B = [1,2,5,7,8,11,20,25,30] and a range value that I would like to assign is 10, then my code should output C = [1,11,25]. (2,5,7,8 were removed because it has the distance less than 10 with the element 1. Next element is 11. 20 is removed because 20 has the distance less than 10 with the element 11. Next is 25 so 30 is removed). You get the idea.
I wrote the code as following:
def RemoveViolations(vec, L):
S = []
P = 0 # pointer
C = 0 # counter
while C < vec.size:
S.append(vec[C])
preC = np.where(vec>S[P]+L)[0]
if preC.size:
C = preC[0]
else:
C = vec.size+1
P = P+1
return np.asarray(S)
So, now, I can do this C = RemoveViolations(B,10), which works like a charm.
Now, the issue is that this is very slow code in python. I have like a sorted vector size of 1 million and it takes some time to finish this code. Is there a better way to do this task?
If I need to implement Cython, how would I change the code to work in C++ environment? I heard it's not really complicated, but a quick search didn't work out well.
Thank you!
The complexity of your algorithm is the problem: Here is a solution in pure python that executes under 0.15s on my 8 years old laptop (your implementation needed 200 seconds; i/e a 1300 times improvement for n=1000000):
import random
def get_filtered_values(dist, seq):
prev_val = seq[0]
compare_to = prev_val + dist
filtered = [prev_val]
for elt in seq[1:]:
if elt <= compare_to: # <-- change to `<` to match desired results;
# this matches the results of your implementation
continue
else:
compare_to = elt + dist
filtered.append(elt)
return filtered
B = [1,2,5,7,8,11,20,25,30]
print(get_filtered_values(10, B))
n = 1000000
C = sorted(list(set([random.randint(0, n) for _ in range(n)])))
get_filtered_values(10, C)
You can cythonize this code, or numpyize it as you wish, but it probably will not be necessary.
My code below checks surrounding pixels to my object's pixel in python.
self.surr = [None, None, None, None, None, None, None, None]
for i in range(9):
#for x in range(-1, 2):
#for y in range(-1, 2):
if i != 5:
x = i % 3 - 2
y = int((i % 3) / 3) - 1
if x == 0 and y == 0:
pass
else:
PAI = allPixels[(self.x + x) % width][(self.y + y) % height] if allPixels[(self.x + x) % width][(self.y + y) % height] != None else None
self.surr[(y * 3) + x] = (PAI)
return self.surr
This returns a list of length 8 that holds either a Pixel object or None. allPixels is a 2D array that also holds wither a Pixel object or None.
I've tried then nested loops that are commented out but they run just a bit slower that the method I'm currently using. This however, is still too slow as if there are 3000 pixels on the screen, which is the lower bounds of the total pixels there will be on the screen in the end, do the maths and you have a LOT going on every frame.
How can I make this run faster using maybe NumPy or some other method?
If you want to see the whole code, it can be found here: https://pastebin.com/EuutUVjS
Thanks for any help you can give me!
What you're doing inherently requires looping—but if you can move that looping into numpy, it'll often get 5-20x faster.
In your case, what you're trying to do is to compare each pixel to its neighbors. How can you do that as an array-wide operation? Simple: compare the array to the same array shifted by 1.
Here's a simpler example:
>>> a = np.array([1,2,4,8,16])
>>> for i in range(1, len(a)):
... print(a[i] - a[i-1], end=' ')
1 2 4 8
>>> print(a[1:] - a[:-1])
[1 2 4 8]
So, for a 2D array, it's just:
north = a[:-1]
ne = a[:-1,1:]
east = a[:,1:]
se = a[1:,1:]
south = a[1:]
sw = a[1:,:-1]
west = a[:,:-1]
nw = a[:-1,:-1]
Note that this isn't wasting a whole lot of time or memory building 8 extra arrays; it's just creating 8 views over the same memory.
See this answer on compsci for an example of using these shifted arrays for a Conway Game of Life simulation.
If you want to handle the borders differently, you may need to "zero-extend" the array, but that's the only complexity you're likely to run into.
However, there's a limit to how much benefit you can get out of numpy if you're storing Python objects in it. Normally, you want to store arrays of numbers.
I don't know what's in your Pixel objects, but let's pretend they're just color values, as three floats. In that case, you can use a 2D array with a structured dtype of three floats, or just a 3D array (row by column by r-g-b), either way using NaN values in place of None.
If you do that, array-wide operations can operate at near-machine-native speeds, including using SIMD operations for data-parallelism. If you don't, only the looping happens at native speeds; the arithmetic within the loop is still just as slow as in non-Numpy Python.
I am trying to create a loop in Python with numpy that will give me a variable "times" with 5 numbers generated randomly between 0 and 20. However, I want there to be one condition: that none of the differences between two adjacent elements in that list are less than 1. What is the best way to achieve this? I tried with the last two lines of code, but this is most likely wrong.
for j in range(1,6):
times = np.random.rand(1, 5) * 20
times.sort()
print times
da = np.diff(times)
if da.sum < 1: break
For instance, for one iteration, this would not be good:
4.25230915 4.36463992 10.35915732 12.39446368 18.46893283
But something like this would be perfect:
1.47166904 6.85610453 10.81431629 12.10176092 15.53569052
Since you are using numpy, you might as well use the built-in functions for uniform random numbers.
def uniform_min_range(a, b, n, min_dist):
while True:
x = np.random.uniform(a, b, size=n)
np.sort(x)
if np.all(np.diff(x) >= min_dist):
return x
It uses the same trial-and-error approach as the previous answer, so depending on the parameters the time to find a solution can be large.
Use a hit and miss approach to guarantee uniform distribution. Here is a straight-Python implementation which should be tweakable for numpy:
import random
def randSpacedPoints(n,a,b,minDist):
#draws n random numbers in [a,b]
# with property that their distance apart is >= minDist
#uses a hit-miss approach
while True:
nums = [a + (b-a)*random.random() for i in range(n)]
nums.sort()
if all(nums[i] + minDist < nums[i+1] for i in range(n-1)):
return nums
For example,
>>> randSpacedPoints(5,0,20,1)
[0.6681336968970486, 6.882374558960349, 9.73325447748434, 11.774594560239493, 16.009157676493903]
If there is no feasible solution this will hang in an infinite loop (so you might want to add a safety parameter which controls the number of trials).
Consider a 100X100 array.
i) Generate an array of several thousand random locations within such an array, e.g. (3,75) and (56, 34).
ii) Calculate how often one of your random locations falls within 15 pixels of any of the (straight) edges.
I am trying to do the above question in order to help me to learn the programming language Python, i am new to programming.
Here is what i have got so far:
from __future__ import division
from pylab import *
import math as m
from numpy import *
from random import randrange
N = 3000
coords_array = array([randrange(100) for _ in range(2 * N)]).reshape(N, 2)
This creates the array of N random locations, and no i am trying to create a loop that will append a 1 to an empty list if x>85 or y>85 or x<15 or y<15, then append a zero to the same empty list if x or y is anything else. Then i would find the sum of the list, which would be my count of how many of the random location fall within the edges.
This is the kind of thing i am trying to do:
coordinate=coords_array[x,y]
b=[]
def location(x,y):
if x>85 or y>85:
b.appnend(1)
if x<15 or y<15:
b.append(1)
else:
b.append(0)
print b
print x
But i am having trouble assigning the array as x and y variables. I want to be able assign each row of the set of random coordinates as an x,y pair so that i can use it in my loop.
But i do not know how to do it!
Please can someone show me how to do it?
Thank you
Ok, the answer to this:
But i am having trouble assigning the array as x and y variables. I
want to be able assign each row of the set of random coordinates as an
x,y pair so that i can use it in my loop
Would be this:
for pair in coords_array:
# Do something with the pair
NumPy arrays behave as regular Python sequences by letting for to iterate over their main axis, meaning pair will contain an array of (in your case) two elements: x and y. You can also do this:
for x, y in coords_array:
# Do something with the pair
NB: I think you wanted to write the function like this:
def location(x,y):
if x>85 or y>85:
b.append(1)
elif x<15 or y<15:
b.append(1)
else:
b.append(0)
or
def location(x,y):
if x>85 or y>85 or x<15 or y<15:
b.append(1)
else:
b.append(0)
or even
def location(x,y):
if not (15 <= x <= 85) or not (15 <= y <= 85):
b.append(1)
else:
b.append(0)
Otherwise, as #TokenMacGuy points out, you'd be inserting two values in certain cases.
NB: from your question I understand you want to write this code specifically to learn Python, but you could do this in a much more straightforward (and efficient) way by just using NumPy functionality
You can let numpy do the looping for you:
n = 3000
coords = np.random.randint(100, size=(n, 2))
x, y = coords.T
is_close_to_edge = (x < 15) | (x >= 85) | (y < 15) | (y >= 85)
count_close_to_edge = np.sum(is_close_to_edge)
Note that the first index of a 100 element array is 0 and the last 99, hence items within 15 positions of the edges are 0...14 and 85...99, hence the >= in the comparison. In the code above, is_close_to_edge is your list, with boolean values.