Use Numpy arrays for large data file - python

I have a numpy array which is of the following type:-
[[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 1 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]]
[[1 1 1 ... 0 0 0]
[1 1 0 ... 0 0 0]
[1 1 1 ... 0 0 0]
...
[0 1 1 ... 1 1 1]
[1 0 0 ... 0 0 0]
[1 0 0 ... 0 0 0]]]
[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]]
[[0 0 0 ... 1 1 0]
[0 0 0 ... 1 0 0]
[0 0 0 ... 1 0 0]
...
[0 0 0 ... 0 0 0]
[1 0 1 ... 0 0 0]
[0 0 0 ... 0 0 0]]]
[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 1 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 0 0]]
[[1 1 1 ... 0 0 0]
[1 1 0 ... 1 1 1]
[0 1 1 ... 1 1 1]
...
[1 0 0 ... 0 0 0]
[1 0 0 ... 1 0 0]
[1 0 0 ... 0 1 0]]]
...
[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 1 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 1 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 1 0 ... 0 0 0]
[1 1 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 1 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 1 0 ... 1 1 0]
...
[0 1 0 ... 0 0 0]
[0 1 0 ... 0 0 0]
[1 0 1 ... 1 1 1]]]
[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 1 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 1 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[1 1 1 ... 1 1 0]
[1 1 0 ... 0 1 1]
[0 0 0 ... 1 0 1]
...
[0 0 1 ... 0 0 0]
[0 1 0 ... 0 0 0]
[1 0 0 ... 0 0 0]]
[[0 0 0 ... 1 0 0]
[1 1 1 ... 0 1 1]
[0 0 1 ... 1 0 1]
...
[1 0 0 ... 1 0 1]
[0 1 0 ... 1 0 0]
[0 0 0 ... 1 0 0]]]
[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 1 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 1 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 1 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[1 1 0 ... 0 0 0]
[1 1 0 ... 0 0 0]
[1 0 0 ... 0 0 0]
...
[1 0 0 ... 0 1 0]
[1 0 0 ... 1 1 0]
[0 1 1 ... 1 1 1]]
[[0 0 0 ... 0 0 0]
[0 1 0 ... 1 1 0]
[1 0 0 ... 0 1 0]
...
[0 0 0 ... 0 0 1]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]]]
I want to extract only 1/4th of the array but the file is too big to be loaded in RAM. I have tried to use mmap but couldn't understand how to do it because it is giving completely different values than that of my numpy array. Even when I loaded the code using the mem map it gives wrong shape and values. I have noticed that on changing dtype the values also change and cannot understand it. My Method for mem map:-
load = np.memmap("dataset/b.npy", mode='r', shape=(14,8,8))

Related

Image fromarray method is not working properly

i created a numpy array of size 10 by 10 with
x = np.eye(10, dtype=np.uint8)
[[1 0 0 0 0 0 0 0 0 0]
[0 1 0 0 0 0 0 0 0 0]
[0 0 1 0 0 0 0 0 0 0]
[0 0 0 1 0 0 0 0 0 0]
[0 0 0 0 1 0 0 0 0 0]
[0 0 0 0 0 1 0 0 0 0]
[0 0 0 0 0 0 1 0 0 0]
[0 0 0 0 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 1 0]
[0 0 0 0 0 0 0 0 0 1]]
then i created a images out of it using fromarray method in pillow lib
im = Image.fromarray(x, mode='1')
im.show()
but the image is completely wrong. Then i converted the image to numpy array
img_arr = np.array(im, dtype=np.uint8)
it looks like this
[[0 0 0 0 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]]
the array is completely wrong.There are no diagonal 1's.
Can some help me ?
complete code
import numpy as np
from PIL import Image, ImageOps
x = np.eye(10, dtype=np.uint8)
print(x)
im = Image.fromarray(x, mode='1')
im.show()
im.save("D:\Desktop/fig.jpeg")
print(im.mode)
img_arr = np.array(im, dtype=np.uint8)
print(img_arr)
Do not use "1". Use "L"
im = Image.fromarray(x, mode='L')
if you really want to use mode "1", you have to encode in bytes. Every bit on each byte is a pixel.
import matplotlib.pyplot as plt
x=np.array(bytes([128,64,32,16,8,4,2,1]))
im = Image.frombytes(data=x, mode='1',size=(8,8))
plt.imshow(im)
plt.show()
x=np.array([0b00010000,
0b00101000,
0b00010000,
0b11111110,
0b00010000,
0b00101000,
0b01000100,
0b01000100],dtype=np.uint8)
im = Image.frombytes(data=x, mode='1',size=(8,8))
plt.imshow(im)
plt.show()

How to add Numpy ndarrays together? [duplicate]

This question already has answers here:
Simple adding two arrays using numpy in python?
(2 answers)
Closed 2 years ago.
I'm trying to add multiple arrays together and I'm stuck.
For example, I have those two arrays:
[[1 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 1 0 1]
[0 0 1 0 0 0 0 1 0 0]
[1 0 0 0 1 0 0 0 0 0]
[1 1 0 0 0 0 0 0 2 0]
[0 0 0 0 0 1 0 0 0 0]
[0 0 0 1 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 1]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 2]]
and
[[1 0 0 0 0 0 0 0 0 0]
[0 1 0 0 1 0 0 0 0 1]
[0 0 1 0 0 0 0 0 0 0]
[0 1 1 0 0 0 0 0 0 0]
[0 0 0 0 2 0 0 0 0 0]
[0 0 0 0 0 1 0 0 0 0]
[1 0 0 0 0 0 1 0 1 0]
[0 1 0 0 0 1 0 1 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]]
How do I do to add them together such that the resulting array will look like this:
element from the 1st row and from the 1st column (top left) in 1st array + element from the 1st row and from the 1st column (top left) in 2nd array = 2
So the element in the 1st row and 1st column in the resulting array will be 2 and so on for every element.
Thanks
Try the .add method for a numpy array:
sum = np.add(firstarray, secondarray)

How to make a generator generate a dataset of tuples (x, y) with x and y being numpy arrays?

I am trying to implement a neural network using keras.fit_generator() function. I have implemented the generator function that yields x, y where x is the data and y is the ground truth for the result and both are (156, 156, 156) numpy arrays. However, when I try to feed the data I get an error that says "Output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: tf.Tensor".
When I checked what I obtain when I create my dataset using the generator function and iterate through it, and what I am getting is indeed a tf.Tensor. However, I couldn't figure out how to make it return a tuple instead of a tf.Tensor with shape (1, 2, 156, 156, 156). What should I do to get a tuple of (x, y)?
For simplicity, I used the following generator function that is supposed to yield (x, y):
def tuple_fun():
for _ in range(10):
x = np.random.rand(156,156,156)
y = np.random.rand(156,156,156)
yield tuple((x, y))
I generated the dataset with the following piece of code:
def dataset_generator(batch_size):
dataset = tf.data.Dataset.from_generator(lambda: tuple_fun(),
output_types=tf.int8,
output_shapes = (2, 156, 156, 156)).batch(batch_size)
return dataset
Then I tried feeding the data into a neural network via keras.fit_generator() as follows:
test_batch_size = 1
test_dataset = dataset_generator(test_batch_size)
iterator = iter(test_dataset)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv3D(input_shape=(156, 156, 156, 1),
filters=5, padding="same", kernel_size=3,
activation='relu',
kernel_initializer=tf.keras.initializers.TruncatedNormal()))
model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(decay=0.002))
model.fit_generator(iterator, steps_per_epoch=1, epochs=1)
And I got the following error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-97-e1055a8161e6> in <module>
26
27 model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(decay=0.002))
---> 28 model.fit_generator(iterator, steps_per_epoch=1, epochs=1)
~/anaconda3/envs/condaEnv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1513 shuffle=shuffle,
1514 initial_epoch=initial_epoch,
-> 1515 steps_name='steps_per_epoch')
1516
1517 def evaluate_generator(self,
~/anaconda3/envs/condaEnv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_generator.py in model_iteration(model, data, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch, mode, batch_size, steps_name, **kwargs)
211 step = 0
212 while step < target_steps:
--> 213 batch_data = _get_next_batch(generator, mode)
214 if batch_data is None:
215 if is_dataset:
~/anaconda3/envs/condaEnv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_generator.py in _get_next_batch(generator, mode)
363 raise ValueError('Output of generator should be '
364 'a tuple `(x, y, sample_weight)` '
--> 365 'or `(x, y)`. Found: ' + str(generator_output))
366
367 if len(generator_output) < 1 or len(generator_output) > 3:
ValueError: Output of generator should be a tuple `(x, y, sample_weight)` or `(x, y)`. Found: tf.Tensor(
[[[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]]
[[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
...
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]]]], shape=(1, 2, 156, 156, 156), dtype=int8)
How can I solve this problem? This was my first question in stackoverflow so I hope I didn't do anything wrong. Thanks in advance for any help!

Save output of python program in text file [duplicate]

This question already has answers here:
Dump a NumPy array into a csv file
(12 answers)
Closed 4 years ago.
I want to save the output of my Python program into a text file to use it in C for some reasons, I don't know how should I do it. The code is:
import networkx as nx
import numpy as np
t_start=0;t_end=1;dt=0.1
tpoints=np.arange(t_start,t_end,dt)
G = nx.grid_2d_graph(20,20, periodic=False, create_using=None)
adj_matrix=nx.adjacency_matrix(G)
print(adj_matrix.todense())
If the number of nodes were less than 20 (like 10 or lower) the output would be:
[[0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0]
[1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0]
[0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0]
[0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0]
[1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0]
[0 1 0 0 1 0 1 0 0 1 0 0 0 0 0 0]
[0 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0]
[0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0]
[0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0]
[0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0]
[0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0]
[0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1]
[0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0]
[0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1]
[0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0]]
But when the number of nodes increase, the output is like this:
[[0 1 0 ... 0 0 0]
[1 0 1 ... 0 0 0]
[0 1 0 ... 0 0 0]
...
[0 0 0 ... 0 1 0]
[0 0 0 ... 1 0 1]
[0 0 0 ... 0 1 0]]
And so I can't copy it manually to a text file. So I need a command to write this matrix completely to a text file. Thanks for your answers.
The displayed output of a numpy array is not meant to display everything.
If you want to save the array to a file, you can use tofile. There you can define if you want a binary or a text file.

assigning values to a matrix in python

I want to store this:
for i in range(len(cpi)):
print((cf[:i+1] / cpi[i]).astype(int))
Into a matrix. I tried this:
payment = np.zeros((len(cpi), len(cpi)))
for i in range(len(cpi)):
payment = cf[:i+1] / cpi[i]).astype(int)
But that gives me the wrong matrix that looks like this:
[[0 1 0 ..., 1 0 3]
[2 9 2 ..., 4 0 4]
[4 0 8 ..., 9 6 3]
...,
[0 0 0 ..., 0 0 0]
[1 3 0 ..., 1 1 1]
[0 3 0 ..., 1 0 0]]
When the output show be this:
[[0 1 0 1 1 1 0 5 1 0 2]]
[[0 0 0 0 0 0 0 2 0 0 1]
[1 3 1 0 0 1 1 1 1 0 1]]
[[0 0 0 0 0 0 0 1 0 0 0]
[0 2 0 0 0 0 0 0 0 0 0]
[1 0 2 0 2 1 1 0 2 1 0]]
[[0 1 0 1 1 1 0 5 1 0 2]
[1 6 1 0 0 1 1 1 2 0 2]
Note that is not all the elements just some of them.
The issue is you are assigning new value to payment variable each time you loop, when you actually want to just append to it. For example (not tested):
payment=[]
for i in range(len(cpi)):
payment.append((cf[:i+1] / cpi[i]).astype(int))

Categories

Resources