i ran into an error while trying to train a tensorflow model. In my code below i can choose a folder with pictures of the person and its name. The Pictures should get trained by the model. At the first dataset it all works finde, but when i am trying to train a second set it gives me the error that the Shapes are incompatible.
I am thankful for every help!
import os
import cv2
import numpy as np
import tensorflow as tf
# User input name and filepath
name = "Tony Stark"
folder_path = "C:/Users/johan/Pictures/tony"
img_size = (160, 160)
# Read labels from the label.txt file
label_dict = {}
if os.path.isfile("label.txt"):
with open("label.txt", "r") as file:
label_dict = eval(file.read())
# Add new label if it does not exist
if name not in label_dict:
new_label = len(label_dict)
label_dict[name] = new_label
with open("label.txt", "w") as file:
file.write(str(label_dict))
else:
new_label = label_dict[name]
# Load the model if it exists, create a new one if it doesn't
if os.path.exists("tony_face_model.h5"):
model = tf.keras.models.load_model("tony_face_model.h5")
print("Loaded model from disk")
else:
model = tf.keras.models.Sequential([
tf.keras.layers.Input(shape=(img_size[0], img_size[1], 3)),
tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation="relu"),
tf.keras.layers.Dense(len(label_dict), activation="softmax"),
])
model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
print("Created a new model")
face_images = []
labels = []
for i, filename in enumerate(os.listdir(folder_path)):
img_path = os.path.join(folder_path, filename)
img = cv2.imread(img_path)
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, scaleFactor=1.2, minNeighbors=5)
for (x, y, w, h) in faces:
if w == h:
face_image = img[y:y + h, x:x + w]
face_image = cv2.resize(face_image, img_size)
face_images.append(face_image)
label = new_label
labels.append(label)
if len(face_images) > 0:
face_images = np.array(face_images)
labels = np.array(labels)
# convert labels to one-hot encoding
labels = tf.keras.utils.to_categorical(labels)
# Split the data into training and validation sets
split = int(len(face_images) * 0.8)
train_images = face_images[:split]
train_labels = labels[:split]
val_images = face_images[split:]
val_labels = labels[split:]
# train the model on the new data
model.fit(x=train_images, y=train_labels, validation_data=(val_images, val_labels), epochs=30)
model.save("tony_face_model.h5")
print("Model saved to disk")
else:
print("No new faces found")
An here the Error-Code:
Loaded model from disk
Epoch 1/30
Traceback (most recent call last):
File "C:\Users\johan\PycharmProjects\recognitionSoft\main.py", line 78, in <module>
model.fit(x=train_images, y=train_labels, validation_data=(val_images, val_labels), epochs=30)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\johan\AppData\Local\Temp\__autograph_generated_file0ma0lsv9.py", line 15, in tf__train_function
retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
ValueError: in user code:
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\engine\training.py", line 1249, in train_function *
return step_function(self, iterator)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\engine\training.py", line 1233, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\engine\training.py", line 1222, in run_step **
outputs = model.train_step(data)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\engine\training.py", line 1024, in train_step
loss = self.compute_loss(x, y, y_pred, sample_weight)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\engine\training.py", line 1082, in compute_loss
return self.compiled_loss(
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\engine\compile_utils.py", line 265, in __call__
loss_value = loss_obj(y_t, y_p, sample_weight=sw)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\losses.py", line 152, in __call__
losses = call_fn(y_true, y_pred)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\losses.py", line 284, in call **
return ag_fn(y_true, y_pred, **self._fn_kwargs)
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\losses.py", line 2004, in categorical_crossentropy
return backend.categorical_crossentropy(
File "C:\Users\johan\PycharmProjects\recognitionSoft\venv\lib\site-packages\keras\backend.py", line 5532, in categorical_crossentropy
target.shape.assert_is_compatible_with(output.shape)
ValueError: Shapes (None, 1) and (None, 2) are incompatible
Process finished with exit code 1
And here the different shape values:
Set1:
train_images
[[[[132 160 174]
[127 159 172]
[123 157 170]
...
[ 23 32 38]
[ 15 23 29]
[ 17 25 31]]
[[131 159 174]
[127 159 172]
[127 161 174]
...
[ 42 48 60]
[ 37 43 56]
[ 21 28 40]]
[[126 156 173]
[123 151 168]
[122 149 166]
...
[ 36 45 55]
[ 54 62 76]
[ 41 47 63]]
...
[[143 172 181]
[144 173 182]
[145 174 183]
...
[ 36 38 40]
[ 27 32 31]
[ 34 41 37]]
[[150 178 188]
[145 174 183]
[134 164 172]
...
[ 31 33 34]
[ 21 26 24]
[ 30 37 31]]
[[142 171 180]
[134 164 172]
[140 169 178]
...
[ 46 48 48]
[ 44 50 45]
[ 42 51 42]]]
[[[244 249 248]
[250 254 253]
[238 244 243]
...
[237 246 242]
[235 244 236]
[241 250 240]]
[[235 240 239]
[244 250 249]
[246 254 253]
...
[235 241 239]
[238 244 240]
[239 244 238]]
[[239 244 243]
[240 245 244]
[240 248 247]
...
[231 237 236]
[237 242 240]
[240 244 239]]
...
[[248 245 240]
[248 245 240]
[243 240 235]
...
[243 246 248]
[241 243 237]
[241 241 232]]
[[248 245 240]
[248 245 240]
[243 240 235]
...
[238 241 247]
[241 244 242]
[239 241 234]]
[[247 244 239]
[246 243 238]
[243 240 235]
...
[226 229 242]
[239 243 244]
[233 237 230]]]
[[[141 150 110]
[141 150 110]
[143 152 112]
...
[ 11 36 53]
[ 5 30 46]
[ 5 30 46]]
[[141 150 110]
[141 150 110]
[143 152 112]
...
[ 11 36 53]
[ 5 30 46]
[ 5 30 46]]
[[147 155 111]
[147 155 111]
[151 159 115]
...
[ 16 39 55]
[ 10 33 47]
[ 10 33 47]]
...
[[192 161 52]
[192 161 52]
[190 159 50]
...
[124 128 132]
[128 132 137]
[128 132 137]]
[[191 160 51]
[191 160 51]
[189 157 49]
...
[122 125 130]
[127 131 136]
[127 131 136]]
[[191 160 51]
[191 160 51]
[189 157 49]
...
[122 125 130]
[127 131 136]
[127 131 136]]]]
train_labels
[[1.]
[1.]
[1.]]
val_images
[[[[ 6 11 16]
[ 6 8 18]
[ 4 5 16]
...
[218 196 161]
[221 196 162]
[222 197 163]]
[[ 7 13 22]
[ 5 8 22]
[ 2 5 19]
...
[230 208 176]
[235 210 178]
[232 209 177]]
[[ 0 8 16]
[ 2 5 19]
[ 1 4 18]
...
[232 210 178]
[234 211 179]
[233 210 178]]
...
[[225 192 148]
[226 192 149]
[225 191 148]
...
[204 170 134]
[204 171 132]
[203 170 131]]
[[225 192 148]
[226 192 149]
[225 191 148]
...
[204 170 135]
[204 171 132]
[202 169 130]]
[[225 192 148]
[226 192 149]
[225 191 148]
...
[205 170 135]
[203 170 131]
[202 169 130]]]]
val_labels
[[1.]]
Set2:
train_images
[[[[213 203 198]
[198 187 183]
[208 198 197]
...
[208 183 163]
[207 183 162]
[206 182 161]]
[[216 206 201]
[206 195 192]
[198 188 187]
...
[208 183 162]
[208 183 162]
[209 182 162]]
[[221 211 205]
[210 199 196]
[177 167 166]
...
[209 183 162]
[208 183 162]
[209 181 161]]
...
[[ 33 4 2]
[ 33 4 2]
[ 34 5 2]
...
[ 64 37 29]
[ 63 36 28]
[ 55 28 20]]
[[ 35 4 2]
[ 35 4 2]
[ 37 4 2]
...
[ 56 29 22]
[ 59 31 25]
[ 53 26 19]]
[[ 35 4 2]
[ 36 4 2]
[ 39 4 2]
...
[ 46 19 12]
[ 54 27 20]
[ 55 28 21]]]
[[[126 50 7]
[127 54 6]
[129 55 8]
...
[126 136 145]
[126 132 135]
[ 89 101 107]]
[[127 52 6]
[125 53 6]
[126 54 6]
...
[122 134 151]
[132 137 143]
[ 59 73 85]]
[[132 58 10]
[130 55 9]
[128 56 8]
...
[100 119 140]
[115 122 136]
[ 93 105 116]]
...
[[159 70 18]
[163 73 18]
[170 80 23]
...
[ 27 22 21]
[ 30 22 19]
[ 31 19 18]]
[[166 77 27]
[162 74 20]
[167 80 24]
...
[ 28 20 20]
[ 27 19 19]
[ 27 19 15]]
[[165 78 22]
[169 82 26]
[168 80 26]
...
[ 29 21 21]
[ 29 22 19]
[ 28 21 18]]]
[[[180 179 175]
[ 56 55 51]
[ 96 95 94]
...
[252 252 247]
[252 252 246]
[252 252 246]]
[[178 177 172]
[ 60 59 55]
[ 83 82 80]
...
[251 250 246]
[251 251 244]
[251 251 244]]
[[227 226 221]
[ 74 73 68]
[ 66 66 64]
...
[251 250 246]
[251 250 246]
[251 250 246]]
...
[[111 122 155]
[108 120 152]
[110 121 154]
...
[203 205 205]
[201 205 207]
[201 205 207]]
[[113 126 158]
[110 123 155]
[109 121 153]
...
[201 206 205]
[200 204 205]
[198 204 205]]
[[116 128 160]
[114 126 158]
[112 124 157]
...
[201 206 205]
[198 204 205]
[198 204 205]]]]
train_labels
[[0. 1.]
[0. 1.]
[0. 1.]]
val_images
[[[[249 248 250]
[249 248 251]
[249 249 249]
...
[249 249 249]
[249 249 249]
[249 249 249]]
[[249 248 250]
[249 248 250]
[251 250 248]
...
[249 249 249]
[249 249 249]
[249 249 249]]
[[249 249 249]
[250 249 249]
[251 251 246]
...
[249 249 249]
[249 249 249]
[249 249 249]]
...
[[240 231 219]
[243 233 221]
[245 235 223]
...
[244 244 230]
[237 236 221]
[241 238 223]]
[[240 231 219]
[243 233 221]
[245 235 223]
...
[242 242 229]
[241 241 226]
[236 234 221]]
[[241 233 218]
[243 234 220]
[244 235 221]
...
[118 118 110]
[240 240 231]
[239 237 226]]]]
val_labels
[[0. 1.]]
Related
I have numpy array which shape is (512,512,3)
It is 512 x 512 image.
I want to show image and save as png with matplotlib
How can I do this???
How should I convert??
[[[ 87 48 39]
[107 43 29]
[101 40 28]
...
[115 107 100]
[115 106 100]
[115 107 102]]
[[ 94 44 30]
[106 38 20]
[ 97 38 23]
...
[114 109 103]
[113 108 103]
[114 106 98]]
[[ 87 41 30]
[ 96 40 32]
[ 92 38 37]
...
[114 110 105]
[114 110 105]
[116 109 98]]
...
[[123 112 112]
[140 120 121]
[135 120 119]
...
[215 191 218]
[221 195 223]
[217 196 214]]
[[127 116 119]
[134 115 115]
[138 123 124]
...
[217 195 220]
[220 199 221]
[215 193 208]]
[[125 118 117]
[127 115 116]
[131 121 123]
...
[215 199 220]
[216 198 217]
[202 179 198]]]
You can try the below snip,
from PIL import Image
import numpy as np
# data is your array
img = Image.fromarray(data, 'RGB')
img.save('my.png')
img.show()
plt.imshow(array)
plt.savefig('filename.png')
I would like to take an image and change the scale of the image, while it is a numpy array.
I would like to do it with native NumPy functions w/o PIL, cv2, SciPy etc
now I have this:
from copy import copy
import numpy as np
from scipy import misc
img = misc.face() # racoon from SciPy(np.ndarray)
img2 = copy(img) # copy of racoon, because misc.face() is Descriptor(?)
img2.shape() # (768, 1024, 3)
Which I need shape = (3072, 4096, 3)
I can do it easy with Pillow
CONVERT_IMAGE = Image.fromarray(img.astype('uint8'), 'RGB')
CONVERT_IMAGE = CONVERT_IMAGE.resize((4096, 3072), Image.NEAREST)
IMAGE_AS_ARRAY = np.asarray(CONVERT_IMAGE)
IMAGE_AS_ARRAY.shape # 3072 4096 3
but I realy need to do this only with NumPy functions w/o other libs
Can you help me ? I'm really weak in NumPy and 3D-arrays
Limited to whole integer upscaling with some scaling factor n and without actual interpolation, you could use np.repeat twice to get the described result:
import numpy as np
# Original image with shape (4, 3, 3)
img = np.random.randint(0, 255, (4, 3, 3), dtype=np.uint8)
# Scaling factor for whole integer upscaling
n = 4
# Actual upscaling (results to some image with shape (16, 12, 3)
img_up = np.repeat(np.repeat(img, n, axis=0), n, axis=1)
# Outputs
print(img[:, :, 1], '\n')
print(img_up[:, :, 1])
Here's some output:
[[148 242 171]
[247 40 152]
[151 131 198]
[ 23 185 144]]
[[148 148 148 148 242 242 242 242 171 171 171 171]
[148 148 148 148 242 242 242 242 171 171 171 171]
[148 148 148 148 242 242 242 242 171 171 171 171]
[148 148 148 148 242 242 242 242 171 171 171 171]
[247 247 247 247 40 40 40 40 152 152 152 152]
[247 247 247 247 40 40 40 40 152 152 152 152]
[247 247 247 247 40 40 40 40 152 152 152 152]
[247 247 247 247 40 40 40 40 152 152 152 152]
[151 151 151 151 131 131 131 131 198 198 198 198]
[151 151 151 151 131 131 131 131 198 198 198 198]
[151 151 151 151 131 131 131 131 198 198 198 198]
[151 151 151 151 131 131 131 131 198 198 198 198]
[ 23 23 23 23 185 185 185 185 144 144 144 144]
[ 23 23 23 23 185 185 185 185 144 144 144 144]
[ 23 23 23 23 185 185 185 185 144 144 144 144]
[ 23 23 23 23 185 185 185 185 144 144 144 144]]
----------------------------------------
System information
----------------------------------------
Platform: Windows-10-10.0.16299-SP0
Python: 3.8.5
NumPy: 1.19.2
----------------------------------------
"Im2col" has already been implemented, Implement MATLAB's im2col 'sliding' in Python, efficiently for 2-D images in Python. I was wondering whether it is possible to extend this to arbitrary N-D images? Many applications involve high-dimensional data (e.g. convolutions, filtering, max pooling, etc.).
So the purpose of this question was really just to post my solution to this problem publicly. I could not seem to find such a solution on Google, so I decided to take a stab at it myself. Turns out the implementation is actually quite simple to extend from "Approach #2" in the post referenced in my question!
Efficient Implementation of N-D "im2col"
def im2col(im, win, strides = 1):
# Dimensions
ext_shp = tuple(np.subtract(im.shape, win) + 1)
shp = tuple(win) + ext_shp
strd = im.strides*2
win_len = np.prod(win)
try:
len(strides)
except:
strides = [strides]*im.ndim
strides = [min(i, s) for i, s in zip(im.shape, strides)]
# Stack all possible patches as an N-D array using a strided view followed by reshaping
col = np.lib.stride_tricks.as_strided(im, shape = shp, strides = strd).reshape(win_len, -1).reshape(-1, *ext_shp)
# Extract patches with stride and reshape into columns
slcs = tuple([slice(None, None, None)] + [slice(None, None, s) for s in strides])
col = col[slcs].reshape(win_len, -1)
return col
Efficient Implementation of N-D "col2im"
def col2im(col, im_shp, win, strides = 1):
# Dimensions
try:
len(strides)
except:
strides = [strides]*len(im_shp)
strides = [min(i, s) for i, s in zip(im_shp, strides)]
# Reshape columns into image
if col.ndim > 1:
im = col.reshape((-1, ) + tuple(np.subtract(im_shp, win)//np.array(strides) + 1))[0]
else:
im = col.reshape(tuple(np.subtract(im_shp, win)//np.array(strides) + 1))
return im
Verification That It Works
Let's define an arbitrary 3-D input:
x = np.arange(216).reshape(6, 6, 6)
print(x)
[[[ 0 1 2 3 4 5]
[ 6 7 8 9 10 11]
[ 12 13 14 15 16 17]
[ 18 19 20 21 22 23]
[ 24 25 26 27 28 29]
[ 30 31 32 33 34 35]]
[[ 36 37 38 39 40 41]
[ 42 43 44 45 46 47]
[ 48 49 50 51 52 53]
[ 54 55 56 57 58 59]
[ 60 61 62 63 64 65]
[ 66 67 68 69 70 71]]
[[ 72 73 74 75 76 77]
[ 78 79 80 81 82 83]
[ 84 85 86 87 88 89]
[ 90 91 92 93 94 95]
[ 96 97 98 99 100 101]
[102 103 104 105 106 107]]
[[108 109 110 111 112 113]
[114 115 116 117 118 119]
[120 121 122 123 124 125]
[126 127 128 129 130 131]
[132 133 134 135 136 137]
[138 139 140 141 142 143]]
[[144 145 146 147 148 149]
[150 151 152 153 154 155]
[156 157 158 159 160 161]
[162 163 164 165 166 167]
[168 169 170 171 172 173]
[174 175 176 177 178 179]]
[[180 181 182 183 184 185]
[186 187 188 189 190 191]
[192 193 194 195 196 197]
[198 199 200 201 202 203]
[204 205 206 207 208 209]
[210 211 212 213 214 215]]]
Let's extract all the patches with a non-uniform window and equal stride:
y = im2col(x, [1, 3, 2], strides = [1, 3, 2])
print(y.T) # transposed for ease of visualization
[[ 0 1 6 7 12 13]
[ 2 3 8 9 14 15]
[ 4 5 10 11 16 17]
[ 18 19 24 25 30 31]
[ 20 21 26 27 32 33]
[ 22 23 28 29 34 35]
[ 36 37 42 43 48 49]
[ 38 39 44 45 50 51]
[ 40 41 46 47 52 53]
[ 54 55 60 61 66 67]
[ 56 57 62 63 68 69]
[ 58 59 64 65 70 71]
[ 72 73 78 79 84 85]
[ 74 75 80 81 86 87]
[ 76 77 82 83 88 89]
[ 90 91 96 97 102 103]
[ 92 93 98 99 104 105]
[ 94 95 100 101 106 107]
[108 109 114 115 120 121]
[110 111 116 117 122 123]
[112 113 118 119 124 125]
[126 127 132 133 138 139]
[128 129 134 135 140 141]
[130 131 136 137 142 143]
[144 145 150 151 156 157]
[146 147 152 153 158 159]
[148 149 154 155 160 161]
[162 163 168 169 174 175]
[164 165 170 171 176 177]
[166 167 172 173 178 179]
[180 181 186 187 192 193]
[182 183 188 189 194 195]
[184 185 190 191 196 197]
[198 199 204 205 210 211]
[200 201 206 207 212 213]
[202 203 208 209 214 215]]
Let's convert this back to a (downsampled) image:
z = col2im(y, x.shape, [1, 3, 2], strides = [1, 3, 2])
print(z)
[[[ 0 2 4]
[ 18 20 22]]
[[ 36 38 40]
[ 54 56 58]]
[[ 72 74 76]
[ 90 92 94]]
[[108 110 112]
[126 128 130]]
[[144 146 148]
[162 164 166]]
[[180 182 184]
[198 200 202]]]
As you can see, the final output is indeed the downsampled image that we expect (you can easily check this by going value by value). The dimensionality and strides I chose were purely illustrative. There's no reason why the window size has to be the same as your stride or that you can't go higher than 3 dimensions.
Applications
If you want to use this practically, all you have to do is intercept the output of im2col before turning it back into an image. For example, if you want to do pooling, you could take the mean or the maximum across the 0th axis. If you want to do a convolution, you just need to multiply this by your flattened convolutional filter.
There may be more efficient alternatives to this already implemented under the hood of Tensorflow, etc. that are faster than "im2col." This is not meant to be the MOST efficient implementation. And of course, you could possibly optimize my code further by eliminating the intermediate reshaping step in "im2col," but it wasn't immediately obvious to me so I just left it at that. If you have a better solution, let me know. Anyways, hope this helps someone else looking for the same answer!
I need to display data from the first column and count the number of appearance of every number:
file Path:
[162 164 168 177 189 190 195 254 255 52]
[152 190 195 74 254 164 249 90 151 52]
[ 47 126 254 152 74 195 164 151 189 52]
[116 120 149 164 152 151 195 189 21 52]
[ 34 195 59 199 252 38 82 189 21 52]
[199 164 151 59 82 38 21 189 227 52]
[ 69 170 38 34 177 153 21 189 52 227]
[ 34 107 177 149 118 21 69 189 52 227]
[ 51 88 75 59 38 107 177 189 52 227]
[109 38 149 112 118 51 177 52 189 227]
[ 89 25 75 59 177 170 107 52 189 227]
[244 107 59 170 88 56 89 52 189 227]
[ 30 183 107 59 170 88 56 52 189 227]
Code:
file="Path"
with open(file) as f:
lines = f.readlines()
result = []
for x in lines:
result.append(x.split(' ')[0])
print(result)
f.close()
The expected results are: 162 152, 47, 116, 34, 199, 69, 34, ...
However, what my code gives me is:
['[162', '[152', '[', '[116', '[', '[199', '[', '[', ...
As you read your input, you need to be a little more careful in processing each line. First, slice off the brackets with x[1:-1] (eliminate the end characters).
Now you can split the line, grab the first field, and convert to an integer:
with open("Path") as f:
result = []
for line in f.readlines():
field1 = line[1:-1].split()[0]
result.append(int(field1))
print(result)
Output:
[162, 152, 47, 116, 34, 199, 69, 34, 51, 109, 89, 244, 30]
You can collapse this to a single statement if you want:
result = [int(line[1:-1].split()[0])
for line in open("Path").readlines()]
To just get the first element of each list:
file="Path"
with open(file) as f:
result = [x[0] for x in f.readlines()]
print(result)
To get counts of those values, use Python's Counter:
from collections import Counter
file="Path"
with open(file) as f:
result = [x[0] for x in f.readlines()]
print(Counter(result))
Having the following 3D array (9,9,9):
np.arange(729).reshape((9,9,9))
[[[ 0 1 2 3 4 5 6 7 8]
[ 9 10 11 12 13 14 15 16 17]
[ 18 19 20 21 22 23 24 25 26]
[ 27 28 29 30 31 32 33 34 35]
[ 36 37 38 39 40 41 42 43 44]
[ 45 46 47 48 49 50 51 52 53]
[ 54 55 56 57 58 59 60 61 62]
[ 63 64 65 66 67 68 69 70 71]
[ 72 73 74 75 76 77 78 79 80]]
...
[[648 649 650 651 652 653 654 655 656]
[657 658 659 660 661 662 663 664 665]
[666 667 668 669 670 671 672 673 674]
[675 676 677 678 679 680 681 682 683]
[684 685 686 687 688 689 690 691 692]
[693 694 695 696 697 698 699 700 701]
[702 703 704 705 706 707 708 709 710]
[711 712 713 714 715 716 717 718 719]
[720 721 722 723 724 725 726 727 728]]]
How do I reshape it to look like this 2D array (27,27):
[[0 1 2 3 4 5 6 7 8 81 82 83 84 85 86 87 88 89 162 163 164 165 166 167 168 169 170]
[9 10 11 12 13 14 15 16 17 90 91 92 93 94 95 96 97 98 171 172 173 174 175 176 177 178 179]
...
[558 559 560 561 562 563 564 565 566 639 640 641 642 643 644 645 646 647 720 721 722 723 724 725 726 727 728]
You can firstly reshape the array to a 4d array, swap the second and third axises and then reshape it to 27 X 27:
a.reshape(3,3,9,9).transpose((0,2,1,3)).reshape(27,27)
#array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 81, 82, 83, 84,
# 85, 86, 87, 88, 89, 162, 163, 164, 165, 166, 167, 168, 169,
# 170],
# [ 9, 10, 11, 12, 13, 14, 15, 16, 17, 90, 91, 92, 93,
# 94, 95, 96, 97, 98, 171, 172, 173, 174, 175, 176, 177, 178,
# 179],
# ...
# [558, 559, 560, 561, 562, 563, 564, 565, 566, 639, 640, 641, 642,
# 643, 644, 645, 646, 647, 720, 721, 722, 723, 724, 725, 726, 727,
# 728]])
import numpy as np
x = np.arange(729).reshape(9, 9, 9)
y = x.transpose(1, 0, 2).reshape(27, 27)
y[y[:,2].argsort()]
Explanations
I used numpy.transpose to permute the stride and shape information for each axis.
>>> x.strides
(324L, 36L, 4L)
>>> x.transpose(1, 0, 2).strides
(36L, 324L, 4L)
more info in this answer.
Then I used numpy.reshape the 3D (9L, 9L, 9L) in 2D as expected
>>> x.reshape(27, 27)
(27L, 27L)
Of course, the combination of functions (like transpose and reshape) is very common in numpy. It allows you do to this matrix transformation in a one-liner:
x.transpose(1, 0, 2).reshape(27, 27)
EDIT
As #PaulPanzer point it out, the array was unsorted.
To sort array column by column, one can use:
y[y[:,2].argsort()]
But maybe it isn't the easiest answer anymore.
If you are ok with moving data in a list you can use:
np.hstack([x for x in np.arange(729).reshape((9,9,9))])