I've written code that collects telemetry information from the device and saves it in a .csv file.
However, I'm not able to detect the GPU from the GPUtil:
import GPUtil
GPU = GPUtil.getAvailable()
print(GPU)
print(GPUtil.__version__)
Output:
[]
1.4.0
I didn't find information if it has any relation with the type of architecture of Jetson.
Below is some additional system information:
After a lot of hard work to come up with some solution, I came to some conclusions that I share here.
Reading a little more about GPUtil and analyzing the GPUtil source code here it becomes clear that:
GPUtil is a Python module for getting the GPU status from NVIDA GPUs using nvidia-smi.
Unfortunately, nvidia-smi does not work on NVIDIA Jetson devices (Xavier NX, Nano, AGX Xavier, TX1, TX2).
As my goal was to fetch telemetry data, there were two options:
tegrastats;
jetson-stats.
I found tegrastats options quite limited as tegrastats reports memory and processor usage for Tegra based devices, but it would be necessary to create a bash file, for, for example, automate the process.
You can find the tegrastats utility here.
As my goal was to create a Python script to fetch telemetry data together with other libraries, like cputil, the solution adopted was use the jtop from jetson-stats as a Python library.
jetson-stats is a package for monitoring and control your NVIDIA Jetson [Xavier NX, Nano, AGX Xavier, TX1, TX2] and works with all NVIDIA Jetson ecosystem.
jtop is a system monitoring utility that runs on the terminal and see and control realtime the status of your NVIDIA Jetson. CPU, RAM, GPU status and frequency and other.
To use it, was needed to install jetson-stats:
$ sudo -H pip install -U jetson-stats
To use jtop just type in the terminal the command jtop. The prompt interface will be show.
To import jtop as a Python library just write the following line of code in Python script:
from jtop import jtop
And in my specific case I used the following snippet code:
with jtop() as jetson:
xavier_nx = jetson.stats
CPU_temperature = xavier_nx['Temp CPU']
GPU_temperature = xavier_nx['Temp GPU']
Thermal_temperature = xavier_nx['Temp thermal']
.stats returns a python dict structure, and the available data values are:
time, uptime, jetson_clocks, nvp model, CPU1, CPU2, CPU3, CPU4, CPU5,
CPU6, GPU, MTS FG, MTS BG, RAM, EMC, SWAP, APE, NVENC, NVDEC, NVJPG,
fan, Temp AO, Temp AUX, Temp CPU, Temp GPU, Temp thermal, power cur,
power avg.
Unfortunately jetson-stats doesn't work with Docker. This can be a negative and important point when thinking about using this service in your application.
jetson-stats also works in Docker.
Following the documentation: https://rnext.it/jetson_stats/docker.html
Install jetson-stats on your host
Install jetson-stats on your container as well
Pass to your container /run/jtop.sock:/run/jtop.sock
example of Docker
docker run --rm -it -v /run/jtop.sock:/run/jtop.sock rbonghi/jetson_stats:latest
I hope I have helped you
Related
I'm trying to use the university server for my deep code, all CPU's core on the server is 64 but I have to use just 24 cores to everybody can use the server too. I try to limit my CPU resource. I search all StackOverflow to find a solution but all suggestion doesn't work for me for example downgrade tensorflow and use
config = tf.ConfigProto(allow_soft_placement=True,
intra_op_parallelism_threads=ncpu,
inter_op_parallelism_threads=ncpu)
and some others solutions by using
import tensorflow as tf
tf.config.threading.set_intra_op_parallelism_threads(numb)
tf.config.threading.set_inter_op_parallelism_threads(numb)
I have to use TensorFlow version 2 or higher because I use 'kerastuner' package in my code
If you have Admin rights on the server and its running a Version of Windows, you can simply restrict the resources via the task-manager.
If you want to do it by code... It looks like its a bug in Tensorflow, which might be fixed, regarding to the github issue.
You might want to try:
export OMP_NUM_THREADS=2
tf.config.threading.set_intra_op_parallelism_threads(2)
tf.config.threading.set_inter_op_parallelism_threads(1)
As this was reported working by Leslie-Fang. If this does not work for you, I guess your only option is to join the github discussion, until its fixed.
My goal is to set up my PC for machine and deep learning through my GPU. I've read about all the different components however I can not connect the dots for what I need to do.
OS: Ubuntu 20.04
GPU: Nvidia RTX 2070 Super
Anaconda: 4.8.3
I've installed the nvidia-cuda-toolkit (10.1.243), but now what?
How does this integrate with jupyter notebook?
The 3 python modules I want to work with are:
turicreate - I've gotten this to run off CPU but not GPU
scikit-learn
tensorflow
matlab
I know cuDNN and pyCUDA fit in there somewhere.
Any help is appreciated. Thanks
First of all - I have the experience limited to ubuntu 18.04 and 16.xx and python DL frameworks. But I hope some sugestions will be helpfull.
If I were familiar with docker I would rather consider to use docker instead of setting-up everything from scratch. This approach is described in section about tensorflow container
If you decided to setup all components yourself please see this guideline
I used some contents from it for 18.04, succesfully.
be carefull with automatic updates. After the configuration is finished and tested protect it from being overwritten with newest version of CUDAor TensorRT.
Answering one of your sub-questions - How does this integrate with jupyter notebook? - it does not, becuase it is unneccesary. CUDA library cooperates with a framework such as Tensorflow, not with the Jupyter. Jupyter is just an editor and execution controller on the server side.
So I am trying to compile TensorFlow from the source (using a clone from their git repo from 2019-01-31). I installed Bazel from their shell script (https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh).
I executed ./configure in the tensorflow code and provided the default settings except for adding my machine specific -m options (-mavx2 -mfma) and pointing python to the correct python3 location (/usr/bin/py3). I then ran the following command as per the tensorflow instructions:
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package //tensorflow:libtensorflow_framework.so //tensorflow:libtensorflow.so
Now that continues to run and run, I haven't seen it complete yet (though I am limited to letting it run for a maximum of about 10 hours). It produces a ton of INFO: warnings regarding signed and unsigned integers and control reaching the end of non-void functions. None of these appear fatal. Compilation continues to tick with the two numbers continuing to grow ('[N,NNN / X,XXX] 4 actions running') and files ticking through 'Compiling'.
The machine is an EC2 instance with ~16GiB of RAM, CPU is 'Intel(R) Xeon(R) CPU E5-2686 v4 # 2.30GHz' with I believe 4-cores, plenty of HDD space (although compilation seems to eat QUITE a bit, > 1GiB)
Any ideas on what's going on here?
Unfortunately, some programs can take a long time to compile. A couple of hours of compilation is not strange for tensorflow on your setup.
There are reports of it taking 50 minutes on a considerably faster machine
A solution to this problem is to use pre-compiled binaries that are available with pip, instructions can be found here: https://www.tensorflow.org/install/pip.html
Basically you can do this:
pip install tensorflow
If you require a specific older version, like 1.15, you can do this:
pip install tensorflow==1.15
For gpu support you add -gpu to the package name, like this:
pip install tensorflow-gpu
And:
pip install tensorflow-gpu==1.15
In development, I have been using the gpu-accelerated tensorflow
https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.2.1-cp35-cp35m-linux_x86_64.whl
I am attempting to deploy my trained model along with an application binary for my users. I compile using PyInstaller (3.3.dev0+f0df2d2bb) on python 3.5.2 to create my application into a binary for my users.
For deployment, I install the cpu version, https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.2.1-cp35-cp35m-linux_x86_64.whl
However, upon successful compilation, I run my program and receive the infamous tensorflow cuda error:
tensorflow.python.framework.errors_impl.NotFoundError:
tensorflow/contrib/util/tensorflow/contrib/cudnn_rnn/python/ops/_cudnn_rnn_ops.so:
cannot open shared object file: No such file or directory
why is it looking for cuda when I've only got the cpu version installed? (Let alone the fact that I'm still on my development machine with cuda, so it should find it anyway. I can use tensorflow-gpu/cuda fine in uncompiled scripts. But this is irrelevant because deployment machines won't have cuda)
My first thought was that somehow I'm importing the wrong tensorflow, but I've not only used pip uninstall tensorflow-gpu but then I also went to delete the tensorflow-gpu in /usr/local/lib/python3.5/dist-packages/
Any ideas what could be happening? Maybe I need to start using a virtual-env..
I've set up pyopencl on my laptop by getting python-pyopencl from multiverse and installing the amd app sdk. To get the Nvidia ICDs I reinstalled the latest Nvidia driver from the driver manager.
My system is a Thinkpad t540p, i7 4700hq, Nvidia gt 730m, 64bit Ubuntu 14.04
To test the opencl installation I ran this pyopencl example: http://wiki.tiker.net/PyOpenCL/Examples/MatrixMultiply
Unfortunately the performance is very bad: Only 2 GFlop/s. Surely the laptop can do better. So I printed the vendor information. It's "GenuineIntel", apparently the kernel is not run on the GPU, but on the CPU. How can I change that ?
It seems like pyopencl doesn't find the GPU.
for dev in ctx.devices:
print dev.vendor
this returns only "GenuineIntel"
The context is created with:
import pyopencl as cl
ctx=cl.create_some_context()
UPDATE:
This seems to be a duplicate of: ERROR: pyopencl: creating context for specific device
There are two issues here.
First, you should specify GPU as the device to execute the kernel on. Replace:
ctx = cl.create_some_context()
with:
platform = cl.get_platforms()
gpus = platform[0].get_devices(device_type=cl.device_type.GPU)
ctx = cl.Context(devices=gpus)
Second, you appear to have Optimus switchable graphics, so the NVIDIA card is actually left in standby and all the graphic tasks are handled by the CPU for powersaving. You will need to activate the discrete GPU for your program by launching it using Bumblebee:
optirun python yourscript.py