Develop Reference

Gigantic memory use in example pytorch program. Why?

I have been trying to debug a program using vast amounts of memory and have distilled it into the following example:
# Caution, use carefully, this can utilise all available memory on your computer
# and render it effectively unresponsive, to the point where you cannot access
# the shell to kill the process; thus requiring reboot.
import numpy as np
import collections
import torch
# q = collections.deque(maxlen=1500) # Uses around 6.4GB
# q = collections.deque(maxlen=3000) # Uses around 12GB
q = collections.deque(maxlen=5000) # Uses around 18GB
def f():
nparray = np.zeros([4,84,84], dtype=np.uint8)
q.append(nparray)
nparray1 = np.zeros([32,4,84,84], dtype=np.float32)
tens = torch.tensor(nparray1, dtype=torch.float32)
while True:
f()
Please note the cautionary message in the 1st line of this program. If you set maxlen to a level where it uses too much of your available RAM, it can crash your computer.
I measured the memory using top (VIRT column), and its memory use seems wildly excessive (details on the commented lines above). From previous experience in my original program if maxlen is high enough it will crash my computer.
Why is it using so much memory?
I calculate the increase in expected memory from maxlen=1500 to maxlen=3000 to be:
4 * 84 * 84 * 15000 / (1024**2) == 403MB.
But we see an increase of 6GB.
There seems to be some sort of interaction between using collections and the tensor allocation as commenting either out causes memory use to be expected; eg commenting out the tensor line leads to total memory use of 2GB which seems much more reasonable.
Thanks for any help or insight,
Julian.

I think PyTorch store and update the computational graph each time you call f(), and thus the graph-size just keeps getting bigger and bigger.
Can you try to free the memory usage by using del(tens) (deleting the reference for the variable after usage), and let me know how it works? (found in PyTorch-documents here: https://pytorch.org/docs/stable/notes/faq.html)

How to free GPU memory in PyTorch

I have a list of sentences I'm trying to calculate perplexity for, using several models using this code:
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
import numpy as np
model_name = 'cointegrated/rubert-tiny'
model = AutoModelForMaskedLM.from_pretrained(model_name).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_name)
def score(model, tokenizer, sentence):
tensor_input = tokenizer.encode(sentence, return_tensors='pt')
repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
masked_input = repeat_input.masked_fill(mask == 1, tokenizer.mask_token_id)
labels = repeat_input.masked_fill( masked_input != tokenizer.mask_token_id, -100)
with torch.inference_mode():
loss = model(masked_input.cuda(), labels=labels.cuda()).loss
return np.exp(loss.item())
print(score(sentence='London is the capital of Great Britain.', model=model, tokenizer=tokenizer))
# 4.541251105675365
Most models work well, but some sentences seem to throw an error:
RuntimeError: CUDA out of memory. Tried to allocate 10.34 GiB (GPU 0; 23.69 GiB total capacity; 10.97 GiB already allocated; 6.94 GiB free; 14.69 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Which makes sense because some are very long. So what I did was to add something like try, except RuntimeError, pass.
This seemed to work until around 210 sentences, and then it just outputs the error:
CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
I found this which had a lot of discussions and ideas, some were regarding potential faulty GPUs? But I know that my GPU works as this exact code works for other models. There's also talk about batch size here, which is why I thought it potentially relates to freeing up memory.
I tried running torch.cuda.empty_cache() to free the memory like in here after every some epochs but it didn't work (threw the same error).
Update:
I filtered sentences with length over 550 and this seems to remove the CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. error.

You need to apply gc.collect() before torch.cuda.empty_cache()
I also pull the model to cpu and then delete that model and its checkpoint. Try what works for you:
import gc
model.cpu()
del model, checkpoint
gc.collect()
torch.cuda.empty_cache()

I don't have an exact answer but I can share some troubleshooting techniques I adopted in similar situations...hope it may be helpful.
First, CUDA error is unfortunately vague sometimes so you should consider running your code on CPU to see if there is actually something else going on (see here )
If the problem is about memory, here are two custom utils I use:
from torch import cuda
def get_less_used_gpu(gpus=None, debug=False):
"""Inspect cached/reserved and allocated memory on specified gpus and return the id of the less used device"""
if gpus is None:
warn = 'Falling back to default: all gpus'
gpus = range(cuda.device_count())
elif isinstance(gpus, str):
gpus = [int(el) for el in gpus.split(',')]
# check gpus arg VS available gpus
sys_gpus = list(range(cuda.device_count()))
if len(gpus) > len(sys_gpus):
gpus = sys_gpus
warn = f'WARNING: Specified {len(gpus)} gpus, but only {cuda.device_count()} available. Falling back to default: all gpus.\nIDs:\t{list(gpus)}'
elif set(gpus).difference(sys_gpus):
# take correctly specified and add as much bad specifications as unused system gpus
available_gpus = set(gpus).intersection(sys_gpus)
unavailable_gpus = set(gpus).difference(sys_gpus)
unused_gpus = set(sys_gpus).difference(gpus)
gpus = list(available_gpus) + list(unused_gpus)[:len(unavailable_gpus)]
warn = f'GPU ids {unavailable_gpus} not available. Falling back to {len(gpus)} device(s).\nIDs:\t{list(gpus)}'
cur_allocated_mem = {}
cur_cached_mem = {}
max_allocated_mem = {}
max_cached_mem = {}
for i in gpus:
cur_allocated_mem[i] = cuda.memory_allocated(i)
cur_cached_mem[i] = cuda.memory_reserved(i)
max_allocated_mem[i] = cuda.max_memory_allocated(i)
max_cached_mem[i] = cuda.max_memory_reserved(i)
min_allocated = min(cur_allocated_mem, key=cur_allocated_mem.get)
if debug:
print(warn)
print('Current allocated memory:', {f'cuda:{k}': v for k, v in cur_allocated_mem.items()})
print('Current reserved memory:', {f'cuda:{k}': v for k, v in cur_cached_mem.items()})
print('Maximum allocated memory:', {f'cuda:{k}': v for k, v in max_allocated_mem.items()})
print('Maximum reserved memory:', {f'cuda:{k}': v for k, v in max_cached_mem.items()})
print('Suggested GPU:', min_allocated)
return min_allocated
def free_memory(to_delete: list, debug=False):
import gc
import inspect
calling_namespace = inspect.currentframe().f_back
if debug:
print('Before:')
get_less_used_gpu(debug=True)
for _var in to_delete:
calling_namespace.f_locals.pop(_var, None)
gc.collect()
cuda.empty_cache()
if debug:
print('After:')
get_less_used_gpu(debug=True)
2.1 free_memory allows you to combine gc.collect and cuda.empty_cache to delete some desired objects from the namespace and free their memory (you can pass a list of variable names as the to_delete argument). This is useful since you may have unused objects occupying memory. For example, imagine you loop through 3 models, then the first one may still take some gpu memory when you get to the second iteration (I don't know why, but I've experienced this in notebooks and the only solution I could find was to either restart the notebook or explicitly free memory). However, I have to say that is not always practical as you need to know which variables are holding GPU memory...and that's not always the case, especially when you have a lot of gradients internally associated with the model (see here for more info). One thing you could also try is to use with torch.no_grad(): instead of with torch.inference_mode():; they should be equivalent but it may be worth a try...
2.2 in case you have a multigpu environment you could consider alternately switching to the less used GPU thanks to the other utils, get_less_used_gpu
Also, you can try to track GPU usage to see when the error happens and debug from there. The best/simplest way I can suggest is using nvtop if you are on a linux platform
Hope something turns out to be useful :)

Tensorflow 2: how to switch execution from GPU to CPU and back?

In tensorflow 1.X with standalone keras 2.X, I used to switch between training on GPU, and running inference on CPU (much faster for some reason for my RNN models) with the following snippet:
keras.backend.clear_session()
def set_session(gpus: int = 0):
num_cores = cpu_count()
config = tf.ConfigProto(
intra_op_parallelism_threads=num_cores,
inter_op_parallelism_threads=num_cores,
allow_soft_placement=True,
device_count={"CPU": 1, "GPU": gpus},
)
session = tf.Session(config=config)
k.set_session(session)
This ConfigProto functionality is no longer available in tensorflow 2.0 (there I'm using the integrated tensorflow.keras). In the beginning, it is possible to run tf.config.experimental.set_visible_devices() in order to e.g. disable the GPU, but any subsequent calls to set_visible_devices result in RuntimeError: Visible devices cannot be modified after being initialized. Is there a way of re-initializing the visible devices or is there another way of switching the devices available?

You can use tf.device to explicitly set which device you want to use. For example:
import tensorflow as tf
model = tf.keras.Model(...)
# Run training on GPU
with tf.device('/gpu:0'):
model.fit(...)
# Run inference on CPU
with tf.device('/cpu:0'):
model.predict(...)
If you only have one CPU and one GPU, the names used above should work. Otherwise, device_lib.list_local_devices() can give you a list of your devices. This post gives a nice function for listing just the names, which I adapt here to also show CPUs:
from tensorflow.python.client import device_lib
def get_available_devices():
local_device_protos = device_lib.list_local_devices()
return [x.name for x in local_device_protos if x.device_type == 'GPU' or x.device_type == 'CPU']

Does using tf.device can help you?
With that, you can set some operations either on CPU or on GPU.

I would just restart the kernel, this worked for me

Initializing variables whose sizes are greater than 2GB in tensorflow

I recently need to initialize a large embedding matrix in tensorflow (greater than 2GB) and the size hits the 2GB limit on protocol buffer.
I searched stackoverflow for help and was able to fine this excellent answer explaining the use of placeholder for this kind of situation. While the placeholder solution works quite well, I found out that the following solution using tf initializer also works, but I don't understand why this solution will not be a problem to the 2GB limit.
import tensorflow as tf
mb = 2 ** 20
gb = 2 ** 30
tf.reset_default_graph()
with tf.Session() as sess:
value = tf.random_uniform([2 * gb]) # using np.random.randn will break the limit
v = tf.get_variable("v", initializer=value, dtype=tf.float32)
sess.run(tf.global_variables_initializer())
print(sess.run(v[:10]))
graph = tf.get_default_graph()
print(graph.as_graph_def().ByteSize() / mb) # 0.0012083053588867188
Does this work because tf.random* initializers internally inspect the shape of the tensors to be generated and switch to a placeholder implementation accordingly? However, if we look at the graph on tensorboard, we will not see any placeholder node in the graph.
Thanks!

Strange error when moving tf.nn.dynamic_rnn to GPU

I'm using to following setup:
Fedora 26, NVIDIA GTX 970, CUDA 8.0, CUDNN 6.0 and tensorflow-gpu==1.3.0 on python
My problem is that, when forcing a dynamic_rnn operator to run on my gpu using:
with tf.name_scope('encoder_both_rnn'),tf.device('/gpu:0'):
_, encoder_state_final_forward = tf.nn.dynamic_rnn(self.encoder_cell_forward,input_ph,dtype=tf.float32,time_major=False,sequence_length=sequence_length,scope='encoder_rnn_forward')
_, encoder_state_final_reverse = tf.nn.dynamic_rnn(self.encoder_cell_reverse,input_reverse,dtype=tf.float32,time_major=False,sequence_length=sequence_length,scope='encoder_rnn_reverse')
i receive the following error when calling the global variable initializer:
InvalidArgumentError: Node 'init/NoOp': Unknown input node '^drawlog_vae_test.DrawlogVaeTest.queue_training/encoder/encoder/encoder_W_mean/Variable/Assign'
The variable is created using the following statement:
self.encoder_W_mean = u.weight_variable([self.intermediate_state_size * 2,self.intermediate_state_size*2],name='encoder_W_mean')
with
def weight_variable(shape,name=None,use_lambda_init=False):
with tf.name_scope(name):
num_weights = float(reduce(lambda x,y: x*y,shape))
initial = tf.truncated_normal(shape,stddev=1) * math.sqrt(2.0/num_weights)
if use_lambda_init:
initial = lambda: np.random.normal(size=shape)
return tf.Variable(initial,dtype=tf.float32)
The strange thing about this is, the variable has nearly nothing to do with the both rnns. Is there any chance running my rnn on the GPU? Or is this just a strange error to tell me I can't run a rnn on a GPU?