I am trying to train the two different models using GPU ID.
I have tried the command CUDA_VISIBLE_DEVICES=1 python filename.py but it picks up the GPU 0 rather than 1.
I also added the os environment variable in my code, but I got the same behavior.
I am not sure how can I fix this as this is first time to use GPU.
Related
I am trying to run a python code on a specific GPU on our server. The server has four GPUs. When I run the code using a virtual environment installed with python 3.8 and tensorflow 2.2, it works correctly on the specific GPU just by adding the below few lines at the first of the script.
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "2" # run the code on a specified GPU
Many threads recommend use the above code to run python scripts on a specific GPU such as here and here.
However, When I tried to use the same way to run another python code on another virtual environment (with lower specifications) that was installed with python version 3.6.9 and tensorflow 1.12, it does not run on the GPU but on the CPU.
How can I run python code on a specific GPU in the case of the second virtual environment?
You can use export CUDA_VISIBLE_DEVICES to define which GPUs are visible to the application. For example, if you want GPUs 0 and 2 visible, use export CUDA_VISIBLE_DEVICES=0,2.
I have two Windows computers, one with and one without a GPU.
I would like to deploy the same python script on both (TensorFlow 1.8 Object Detection), without changing the packaged version of TensorFlow. In other words, I want to run tensorflow-gpu on a CPU.
In the event where my script cannot detect nvcuda.dll, I've tried using a Session config to disable the GPU, like so:
config = tf.ConfigProto(
device_count = {'GPU': 0}
)
and:
with tf.Session(graph=detection_graph, config=config) as sess:
However, this is not sufficient, as TensorFlow still returns the error:
ImportError: Could not find 'nvcuda.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable.
Typically it is installed in 'C:\Windows\System32'. If it is not present, ensure that you have a CUDA-capable GPU with the correct driver installed.
Is there any way to disable checking for a GPU/CUDA entirely and default to CPU?
EDIT: I have read the year-old answer regarding tensorflow-gpu==1.0 on Linux posted here, which suggests this is impossible. I'm interested to know if this is still how tensorflow-gpu is compiled, 9 versions later.
I have two GPUs and would like to run two different networks via ipynb simultaneously, however the first notebook always allocates both GPUs.
Using CUDA_VISIBLE_DEVICES, I can hide devices for python files, however I am unsure of how to do so within a notebook.
Is there anyway to hide different GPUs in to notebooks running on the same server?
You can set environment variables in the notebook using os.environ. Do the following before initializing TensorFlow to limit TensorFlow to first GPU.
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0"
You can double check that you have the correct devices visible to TF
from tensorflow.python.client import device_lib
print device_lib.list_local_devices()
I tend to use it from utility module like notebook_util
import notebook_util
notebook_util.pick_gpu_lowest_memory()
import tensorflow as tf
You can do it faster without any imports just by using magics:
%env CUDA_DEVICE_ORDER=PCI_BUS_ID
%env CUDA_VISIBLE_DEVICES=0
Notice that all env variable are strings, so no need to use ". You can verify that env-variable is set up by running: %env <name_of_var>. Or check all of them with %env.
You can also enable multiple GPU cores, like so:
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0,2,3,4"
I have two GPUs and would like to run two different networks via ipynb simultaneously, however the first notebook always allocates both GPUs.
Using CUDA_VISIBLE_DEVICES, I can hide devices for python files, however I am unsure of how to do so within a notebook.
Is there anyway to hide different GPUs in to notebooks running on the same server?
You can set environment variables in the notebook using os.environ. Do the following before initializing TensorFlow to limit TensorFlow to first GPU.
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0"
You can double check that you have the correct devices visible to TF
from tensorflow.python.client import device_lib
print device_lib.list_local_devices()
I tend to use it from utility module like notebook_util
import notebook_util
notebook_util.pick_gpu_lowest_memory()
import tensorflow as tf
You can do it faster without any imports just by using magics:
%env CUDA_DEVICE_ORDER=PCI_BUS_ID
%env CUDA_VISIBLE_DEVICES=0
Notice that all env variable are strings, so no need to use ". You can verify that env-variable is set up by running: %env <name_of_var>. Or check all of them with %env.
You can also enable multiple GPU cores, like so:
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0,2,3,4"
I am trying to run a Keras script on an AWS instance. While the script runs fine on my own computer (Python 2.7 - no CPU) it causes an error on AWS. I have installed the latest version of Theano - and other scripts (e.g. the mnist tutoral) do not give errors. The script that is causing the issue is a standard Keras tutoral script (https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py). The error is copied in per below (apologies - there might be a better way to capture errors straight from the command line). Any help much appreciated
First page of error message:
End of error message (i have not copied in the entire stack of keras/layers errors)
Somehow you're passing a symbolic value for the border_mode parameter. If this works fine on CPU but not on GPU then, for some reason, the CPU version of the code supports symbolic border modes but the GPU version does not.
If you can, change the border_mode parameter value to be a Python literal instead of a Theano symbolic variable.