I have a sentiment classifier build with Keras that I want to run using my GPU. As Tensorflows GPU support page recommends, I have installed Docker and downloaded a Tensorflow Docker image.
Now when I try to run my code on one of the Tensorflow Images, I get error codes when trying to import stuff like Keras or Pandas.
I am a bit of a newbie when it comes to Docker but as I understand it, the images simply don't have those libraries installed. So what do I do if I want to use additional besides Tensorflow or whatever else is installed on the image? How do I add these to the image?
Option 1: Add packages to the container:
docker exec <container_name> pip install ...
The downside is that you will have to repeat this every time you recreate the container.
Option 2: Create your own image, using tensorflow image as a base
Create a file named Dockerfile:
FROM tensorflow/tensorflow:latest-gpu-jupyter # change if necessary
RUN pip install ...
# Visit https://docs.docker.com/engine/reference/builder/ for format reference
Then build an image from it:
cd /directory/with/the/Dockerfile
docker build -t my-tf-image .
Then run using your own image:
docker run --gpus all -d -v /some/data:/data my-tf-image
I also recommend using docker-compose for dev environment so that you don't have to remember all these commands. You can create a docker-compose.yml and describe the container using YAML format. Then you can just docker-compose build to build and docker-compose up to run.
Related
I'm trying to build a Python app (using streamlit) and getting
Compiled slug size: 664M is too large (max is 500M).
However, when I run
heroku run bash -a pitchcast
$ du -ha --max-depth 1 /app
the only thing that shows up is
4.0K /app
Where are the large files and what can I do to reduce them?
Answering my own question: it seems like the size was due to the Python libraries, especially tensorflow. Switching from tensorflow to tensorflow-cpu got it under 500M
I want to create a version of my Colab notebook that will be immune to changes in the standardly used versions of python and pytorch within colab. Essentially creating something like a docker that will not need to be updated. Is this possible?
Ideally I'd like to keep them as:
Python version: 3.7
PyTorch version: 1.10.0+cu111
CUDA version: 11.1
cuDNN version: 8005
Is this possible?
I don't know anything about Google Colab, but in general, you can achieve this by "pin" versions of:
docker image you built and always go off your custom docker image (meaning any changes to python and packages versions won't affect already built image) link
pin your packages inside Dockerfile so rebuilds ALWAYS use the same version
This could solve your use-case
Let me start by saying I am a beginner on Deep Learning and trying to find my way by following the Tensorflow tutorial, which is mainly applying the inception V3 method to the flowers data set.
https://www.tensorflow.org/tutorials/image_retraining
which includes the following :
cd ~
curl -O (flower data link) -- runs fine
tar xzf flower_photos.tgz --runs fine
bazel build tensorflow/examples/image_retraining:retrain --error: no bazel command found
In order to be able to follow this tutorial, I have also completed the Tensorflow installation tutorial and modified (replaced 35 to 36) it for Python 3.6 compatible whl as follows: pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.2.1-cp36-cp36m-win_amd64.whl
Now back to the main question: After installing the flower data set and installing the bazel package, cygwin64. I went into the Bazel folder and ran the configure file as suggested in the forums as well as touch WORKSPACE and bagel build. When I run the command "bazel build tensorflow/examples/image_retraining:retrain" I still get the error: "Bazel command not found"
I followed similar questions on stackoverflow before openning up my own question, such as: questions- 41791171/bazel-build-for-tensorflow-inception-model and git clone'd the entire Tensorflow folder as instructed but resulted an eror of :bagel: command not found
To summarize, how can I run the Tensorflow Flowerset tutorial and overcome the errors of :bagel: command not found and :bazel: command not found?
It's not mandatory to use Bazel for the TensorFlow Image Retraining tutorial.
You can also run the retrain.py located in the \tensorflow\examples\image_retraining\ folder cloned from the TensorFlow GitHub repo to retrain the Inception v3 model or Mobilenet model.
https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/examples/image_retraining/retrain.py
Put the Flowers datasets folder (flower_photos) under the image_retraining and run the retrain.py as below:
python retrain.py --image_dir flower_photos
You should see the script will download the Inception v3 model.
The image retraining in progress.
After the retraining is completed, you should see the below:
Copy both output_graph.pb and output_labels.txt in the C:\tmp folder, which are the retrain outputs to the image_retraining folder.
To verify the retrained model, you can run the label_image.py as below.
It should show the top 5 predictions.
python label_image.py --image=flower_photos\daisy\21652746_cc379e0eea_m.jpg --graph=output_graph.pb --labels=output_labels.txt
The expected output should be as below:
I have retrained Tensorflow's Inception V3's last layer on a flower dataset. This was done using:
bazel-bin/tensorflow/examples/image_retraining/retrain --image_dir ~/flower_photos
The training was successful, and then I ran:
bazel build tensorflow/examples/label_image:label_image
This too ran fine and next I ran:
bazel-bin/tensorflow/examples/label_image/label_image \
--graph=/tmp/output_graph.pb --labels=/tmp/output_labels.txt \
--output_layer=final_result \
--image=$HOME/flower_photos/daisy/21652746_cc379e0eea_m.jpg
When I run this, I receive an error that says
E tensorflow/examples/label_image/main.cc:285] Not found: Failed to load compute graph at '/tmp/output_graph.pb'
Any help is greatly appreciated, thank you.
I have run this using following command instead of bazel and I found it easier.
python /path_to_file/label_image.py /path_to_image/image.jpeg
First make sure that graph is created after you run retrain.py and it is at the correct location. (default is inside /tmp/). If you want you can get the script here and you can change the file locations as per your need.
Note: If graph file is not created you may want to check this regarding running retrain.py
When I follow the tutorials of "How to Retrain Inception's Final Layer for New Categories", I need to build the retainer like this
bazel build tensorflow/examples/image_retraining:retrain
However, my tensorflow on windows does not have such directory. I am wondering why and how can I solve the problem?
Thank you in advance
In my case tensorflow version is 1.2 and corresponding retrain.py is here.
Download and extract flowers images from here.
Now run the the retrain.py file as
python retrain.py --image_dir=path\to\dir\where\flowers\images\where\extracted --output_lables=retrained_labels.txt --output_graph=retrained_graph.pb
note: the last two arguments in the above command are optional.
Now to test the retrained model:
go the master branch and download the label_image.py code as shown below
Then run python label_image.py --image=image/path/to/test/classfication --graph=retrained_graph.pb --labels=retrained_labels.txt
The result will be like
From the screenshot, it appears that you have installed the TensorFlow PIP package, whereas the instructions in the image retraining tutorial assume that you have cloned the Git repository (and can use bazel to build TensorFlow).
However, fortunately the script (retrain.py) for image retraining is a simple Python script, which you can download and run without building anything. Simply download the copy of retrain.py from the branch of the TensorFlow repository that matches your installed package (e.g. if you've installed TensorFlow 0.12, you can download this version), and you should be able to run it by typing python retrain.py at the Command Prompt.
I had the same problem on windows. My windows could not find script.retrain. I downloaded retrain.py file from tensoflow website at here. Then, copied the file in the tensorflow folder and run the retrain script using Python command.