Develop Reference

Install OpenCV with CUDA in Docker but with very little memory

For a Docker container I am making a build of OpenCV which should work together with CUDA.
Here is an excerpt from the Dockerfile:
ARG OPENCV_VERSION=4.7.0
RUN cd /opt/ &&\
# Download and unzip OpenCV and opencv_contrib and delte zip files
wget https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip &&\
unzip $OPENCV_VERSION.zip &&\
rm $OPENCV_VERSION.zip &&\
wget https://github.com/opencv/opencv_contrib/archive/$OPENCV_VERSION.zip &&\
unzip ${OPENCV_VERSION}.zip &&\
rm ${OPENCV_VERSION}.zip &&\
# Create build folder and switch to it
mkdir /opt/opencv-${OPENCV_VERSION}/build && cd /opt/opencv-${OPENCV_VERSION}/build &&\
# Cmake configure
cmake \
-DOPENCV_EXTRA_MODULES_PATH=/opt/opencv_contrib-${OPENCV_VERSION}/modules \
-DWITH_CUDA=ON \
-DCMAKE_BUILD_TYPE=RELEASE \
# Install path will be /usr/local/lib (lib is implicit)
-DCMAKE_INSTALL_PREFIX=/usr/local \
.. &&\
# Make
make -j"$(nproc)" && \
# Install to /usr/local/lib
make install && \
ldconfig &&\
# Remove OpenCV sources and build folder
rm -rf /opt/opencv-${OPENCV_VERSION} && rm -rf /opt/opencv_contrib-${OPENCV_VERSION}
So far this works, but I have to create a very small container. If I now want to use OpenCV with CUDA, I end up with 2.7GB with all the data, etc.
I would now like to get this smaller. My idea would be to omit the extensions of OpenCV, but then I can no longer create CUDA with OpenCV. Since there are probably libraries in there which are needed.
Is there a way to do this without the libraries or can I delete them and "tell" OpenCV not to look for them? If I just leave them out there are a lot of error messages in OpenCV.
Here is the whole Dockerfile, it is not perfect:
FROM ghcr.io/ifm/ifm3d:latest-l4t-arm64 AS buildstage
COPY requirements.txt /tmp/
COPY python /tmp/python
USER root
ARG JETPACK_VERSION_BASE="r32.4"
ARG JETPACK_VERSION="${JETPACK_VERSION_BASE}.3"
ARG BASE_IMAGE="nvcr.io/nvidia/l4t-base:${JETPACK_VERSION}"
ARG SOC="t186"
ADD --chown=root:root https://repo.download.nvidia.com/jetson/jetson-ota-public.asc /etc/apt/trusted.gpg.d/jetson-ota-public.asc
RUN chmod 644 /etc/apt/trusted.gpg.d/jetson-ota-public.asc \
&& apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& echo "deb https://repo.download.nvidia.com/jetson/common ${JETPACK_VERSION_BASE} main" > /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
&& echo "deb https://repo.download.nvidia.com/jetson/${SOC} ${JETPACK_VERSION_BASE} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
&& cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
&& apt-get update \
&& rm -rf /var/lib/apt/lists/*
ARG CUDA=10.2
ENV CUDA=${CUDA}
ENV PATH /usr/local/cuda-$CUDA/bin:/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/cuda-$CUDA/targets/aarch64-linux/lib:${LD_LIBRARY_PATH}
RUN ldconfig
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES all
ARG OPENCV_VERSION=4.7.0
RUN apt-get update && \
apt-get install -y \
bc \
bzip2 \
language-pack-en-base \
python3-distutils \
python3-pip \
build-essential \
cuda-libraries-dev-${CUDA} \
cuda-cudart-dev-${CUDA} \
cuda-compiler-${CUDA} \
libnvinfer-samples \
ca-certificates \
python-dev \
git \
cmake \
wget \
unzip \
yasm \
pkg-config \
libswscale-dev \
libtbb2 \
libtbb-dev \
libjpeg-dev \
libpng-dev \
libtiff-dev \
libavformat-dev \
libpq-dev \
libxine2-dev \
libglew-dev \
libtiff5-dev \
zlib1g-dev \
libjpeg-dev \
libavcodec-dev \
libavformat-dev \
libavutil-dev \
libpostproc-dev \
libswscale-dev \
libeigen3-dev \
libtbb-dev \
libgtk2.0-dev \
pkg-config \
## Python
python3-numpy \
libopencv-dev \
g++ \
libcudnn8 \
nvidia-cudnn8 \
libsm6 \
libxrender-dev \
&& rm -rf /var/lib/apt/lists/*
USER ifm
RUN python3 -m pip install --upgrade pip
RUN pip3 install --requirement /tmp/requirements.txt
USER root
ARG OPENCV_VERSION=4.7.0
RUN cd /opt/ &&\
# Download and unzip OpenCV and opencv_contrib and delte zip files
wget https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip &&\
unzip $OPENCV_VERSION.zip &&\
rm $OPENCV_VERSION.zip &&\
wget https://github.com/opencv/opencv_contrib/archive/$OPENCV_VERSION.zip &&\
unzip ${OPENCV_VERSION}.zip &&\
rm ${OPENCV_VERSION}.zip &&\
# Create build folder and switch to it
mkdir /opt/opencv-${OPENCV_VERSION}/build && cd /opt/opencv-${OPENCV_VERSION}/build &&\
# Cmake configure
cmake \
-DOPENCV_EXTRA_MODULES_PATH=/opt/opencv_contrib-${OPENCV_VERSION}/modules \
-DWITH_CUDA=ON \
-DCMAKE_BUILD_TYPE=RELEASE \
# Install path will be /usr/local/lib (lib is implicit)
-DCMAKE_INSTALL_PREFIX=/usr/local \
.. &&\
# Make
make -j"$(nproc)" && \
# Install to /usr/local/lib
make install && \
ldconfig &&\
# Remove OpenCV sources and build folder
rm -rf /opt/opencv-${OPENCV_VERSION} && rm -rf /opt/opencv_contrib-${OPENCV_VERSION}
#RUN ls /usr/local/lib/libopencv*
FROM ghcr.io/ifm/ifm3d:latest-l4t-arm64
USER root
#COPY --from=buildstage /usr/local/include/opencv4/ /usr/local/include/opencv4/
COPY --from=buildstage /home/ifm/venv/lib/python3.9/site-packages /home/ifm/venv/lib/python3.9/site-packages
COPY --from=buildstage /usr/local/lib/python3.9/site-packages/cv2/ /home/ifm/venv/lib/python3.9/site-packages/cv2/
COPY --from=buildstage /usr/local/lib/libopencv* /usr/local/lib/
COPY --from=buildstage /usr/lib/x86_64-linux-gnu/ /usr/lib/x86_64-linux-gnu/
#RUN ldconfig
COPY python /tmp/python
The basis container it's a l4t for TX2
Several try to build the container, but currently only OpenCV works without CUDA

Build Python and Pip from source and "bad interpreter: No such file or directory"

I am trying to build Python and Pip from source. Following is the Dockerfile for it:
FROM buildpack-deps:stretch
ARG PYTHON_VERSION="3.6.5"
ARG PIP_VERSION="20.2"
ARG INSTALLATION_PREFIX="/opt/python/${PYTHON_VERSION}"
WORKDIR /usr/src/python
RUN wget -q https://www.python.org/ftp/python/$PYTHON_VERSION/Python-${PYTHON_VERSION}.tgz \
&& apt-get update \
&& apt-get install -y \
make \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
wget \
curl \
llvm \
libncurses5-dev \
libncursesw5-dev \
xz-utils \
tk-dev
RUN tar xvf Python-${PYTHON_VERSION}.tgz \
&& rm -f Python-${PYTHON_VERSION}.tgz
RUN cd Python-$PYTHON_VERSION \
&& ./configure \
--prefix=${INSTALLATION_PREFIX} \
--build=$(dpkg-architecture --query DEB_BUILD_GNU_TYPE) \
--enable-loadable-sqlite-extensions \
--enable-shared \
--with-system-expat \
--with-system-ffi \
--without-ensurepip \
&& make -j $(nproc) \
&& make altinstall
ENV LD_LIBRARY_PATH="${INSTALLATION_PREFIX}/lib:$LD_LIBRARY_PATH"
RUN wget -q https://bootstrap.pypa.io/get-pip.py -O /get-pip.py \
&& /usr/src/python/Python-3.6.5/python /get-pip.py \
--prefix ${INSTALLATION_PREFIX} \
--disable-pip-version-check \
--no-cache-dir \
--no-warn-script-location \
pip==${PIP_VERSION}
WORKDIR /
RUN mv ${INSTALLATION_PREFIX} /tmp/test
ENV LD_LIBRARY_PATH="/tmp/test/lib:$LD_LIBRARY_PATH"
ENV PATH="/tmp/test/bin:$PATH"
When I build the above Dockerfile, it builds fine and I am able to do python3.6 --version and pip --version and I get the expected results.
Now because of a requirement that I have(Imagine building different Python versions from source in a docker image and store these built runtimes somewhere and install them on demand to a location before using it), I need to move this built Python and Pip to a different location, for example /tmp/test...I was hoping that Python and Pip executables would work just as before but when I do a pip --version I get the following error:
root#12c243458190:/# pip --version
bash: /tmp/test/bin/pip: /opt/python/3.6.5/bin/python3.6: bad interpreter: No such file or directory
pip seems to have hard-coded the location of Python to the installation prefix it was given when it was built.
Any ideas on how to fix this issue?

Install Latex on Python AWS Lambda Function using Serverless Framework

I have a Python Lambda function that should create a PDF report. I tried to add a custom Dockerfile to install Latex at Lambda similar to https://github.com/samoconnor/lambdalatex 1
But the function cannot find the installed Latex files.
(The copied file template.tex at the end of the Dockerfile is there)
My serverless.yaml:
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: non-linux
dockerFile: Dockerfile
The Dockerfile:
FROM lambci/lambda:build-python3.6
# The TeXLive installer needs md5 and wget.
RUN yum -y install perl-Digest-MD5 && \
yum -y install wget \
yum -y install latexmk
RUN mkdir /var/src
WORKDIR /var/src
# Download TeXLive installer.
ADD http://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz /var/src/
#COPY install-tl-unx.tar.gz /var/src/
# Minimal TeXLive configuration profile.
COPY texlive.profile /var/src/
# Intstall base TeXLive system.
RUN tar xf install*.tar.gz
RUN cd install-tl-* && \
./install-tl --profile ../texlive.profile
ENV PATH=/var/task/texlive/2017/bin/x86_64-linux/:$PATH
# Install extra packages.
RUN tlmgr install xcolor \
tcolorbox \
pgf \
environ \
trimspaces \
etoolbox \
booktabs \
lastpage \
pgfplots \
marginnote \
tabu \
varwidth \
makecell \
enumitem \
setspace \
xwatermark \
catoptions \
ltxkeys \
framed \
parskip \
endnotes \
footmisc \
zapfding \
symbol \
lm \
sectsty \
stringstrings \
koma-script \
multirow \
calculator \
adjustbox \
xkeyval \
collectbox \
siunitx \
l3kernel \
l3packages \
helvetic \
charter \
latexmk
# Install latexmk.
RUN tlmgr install latexmk
# Remove LuaTeX.
RUN tlmgr remove --force luatex
# Remove large unneeded files.
RUN rm -rf /var/task/texlive/2017/tlpkg/texlive.tlpdb* \
/var/task/texlive/2017/texmf-dist/source/latex/koma-script/doc \
/var/task/texlive/2017/texmf-dist/doc
RUN mkdir -p /var/task/texlive/2017/tlpkg/TeXLive/Digest/ && \
mkdir -p /var/task/texlive/2017/tlpkg/TeXLive/auto/Digest/MD5/ && \
cp /usr/lib64/perl5/vendor_perl/Digest/MD5.pm \
/var/task/texlive/2017/tlpkg/TeXLive/Digest/ && \
cp /usr/lib64/perl5/vendor_perl/auto/Digest/MD5/MD5.so \
/var/task/texlive/2017/tlpkg/TeXLive/auto/Digest/MD5
FROM lambci/lambda:build-python3.6
WORKDIR /var/task
ENV PATH=/var/task/texlive/2017/bin/x86_64-linux/:$PATH
ENV PERL5LIB=/var/task/texlive/2017/tlpkg/TeXLive/
COPY template.tex /var/task/
The Python requirements from the requirements.txt are working perfectly.
Here is the output of the deploy from the Docker part:
Serverless: Building custom docker image from Dockerfile…
Serverless: Docker Image: sls-py-reqs-custom
Do you have any idea what I am doing wrong?

Create Docker container with Nodev4.4.7 and Python3

Trying to create a docker image that has Python3 and Node v4.4.7 so that I can use it as a container for my project that needs both Python and that version of Node.
# Pull base image.
FROM python:3-onbuild
CMD [ "python", "./hello.py" ]
# Install Node.js
RUN \
cd /tmp && \
wget http://nodejs.org/dist/v4.4.7/node-v4.4.7.tar.gz && \
tar xvzf node-v4.4.7.tar.gz && \
rm -f node-v4.4.7.tar.gz && \
cd node-v* && \
./configure && \
CXX="g++ -Wno-unused-local-typedefs" make && \
CXX="g++ -Wno-unused-local-typedefs" make install && \
cd /tmp && \
rm -rf /tmp/node-v* && \
npm install -g npm && \
print '\n# Node.js\nexport PATH="node_modules/.bin:$PATH"' >> /root/.bashrc
# Define working directory.
WORKDIR /data
# Define default command.
CMD ["bash"]
When I first tried it complained about not having a python script to run so added a basic python file: hello.py
that just has this:
print "Hello, Python!"
Then it complains about not having a requirements.txt file so added an empty requirements.txt
Now when I run docker build -t isaacweathersnet/sampledockerimage . it snafus during the node install with
node-v4.4.0/benchmark/arrays/zero-int.js
File "./configure", line 446
'''
^
SyntaxError: Missing parentheses in call to 'print'
The command '/bin/sh -c cd /tmp && wget http://nodejs.org/dist/v4.4.7/node-v4.4.7.tar.gz && tar xvzf node-v4.4.7.tar.gz && rm -f node-v4.4.7.tar.gz && cd node-v* && ./configure && CXX="g++ -Wno-unused-local-typedefs" make && CXX="g++ -Wno-unused-local-typedefs" make install && cd /tmp && rm -rf /tmp/node-v* && npm install -g npm && print '\n# Node.js\nexport PATH="node_modules/.bin:$PATH"' >> /root/.bashrc' returned a non-zero code: 1

Found the solution on Github that had Python and Node. No luck with Python 3+ but worked well with 2.7
https://github.com/nsdont/python-node/blob/master/Dockerfile
FROM python:2.7
RUN \
cd /tmp && \
wget http://nodejs.org/dist/v4.4.7/node-v4.4.7.tar.gz && \
tar xvzf node-v4.4.7.tar.gz && \
rm -f node-v4.4.7.tar.gz && \
cd node-v* && \
./configure && \
CXX="g++ -Wno-unused-local-typedefs" make && \
CXX="g++ -Wno-unused-local-typedefs" make install && \
cd /tmp && \
rm -rf /tmp/node-v* && \
npm install -g npm && \
echo -e '\n# Node.js\nexport PATH="node_modules/.bin:$PATH"' >> /root/.bashrc
# Define working directory.
WORKDIR /data
# Define default command.
CMD ["bash"]

There are nodejs-python and python-nodejs (which is built on top of nodejy-python). It's worth to have a look into there.
python-nodejs provides Node 10.x, npm 6.x, yarn stable, Python latest, pip latest and pipenv latest. The versions used should be adjustable to your version needs. Use the Dockerfile as basis and adjust the RUN section
RUN \
echo "deb https://deb.nodesource.com/node_10.x stretch main" > /etc/apt/sources.list.d/nodesource.list && \
wget -qO- https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list && \
wget -qO- https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
apt-get update && \
apt-get install -yqq nodejs yarn && \
pip install -U pip && pip install pipenv && \
npm i -g npm#^6 && \
rm -rf /var/lib/apt/lists/*
to the Node version you need. The yarn (dependency management alternative to nmp) and in case you need yarn) part can be removed.

Error installing pyV8 from source on ubuntu

when trying to install PyV8 in ubuntu, and type the command:
python setup.py build
then it display this error:
error: command 'c++' failed with exit status 1
anybody have solution about this?

Here is what I have in my Dockerfile. The following is tested and runs in production on top of Debian Stretch. I recommend using exactly the PyV8 / V8 setup that I'm using - I've spent at least a week to figure out which combination doesn't lead to memory leaks. I also recommend reading through the discussion and the JSContext fix here and here.
In short, support for PyV8 is almost non-existent - either you use it just as a toy, or you follow exactly this recipe, or you spend a significant amount of time and effort to fork the repo and make it better. If starting fresh, I recommend using Node-JS instead and communicate through some IPC method with Python.
ENV MY_HOME /home/forge
ENV MY_LIB $FORGE_HOME/lib
# preparing dependencies for V8 and PyV8
ENV V8_HOME $MY_LIB/v8
RUN apt-get update && \
apt-get install -y libboost-thread-dev \
libboost-all-dev \
libboost-dev \
libboost-python-dev \
autoconf \
libtool \
systemtap \
scons
# compiling an older version of boost, required for this version of V8
RUN mkdir -p $MY_LIB/boost && cd $MY_LIB/boost && \
wget http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz && tar -xvzf boost_1_54_0.tar.gz && cd $MY_LIB/boost/boost_1_54_0 && \
./bootstrap.sh && \
./b2 install --prefix=/usr/local --with-python --with-thread && \
ldconfig && \
ldconfig /usr/local/lib
# preparing gcc 4.9 - anything newer will lead to errors with the V8 codebase
ENV CC "gcc-4.9"
ENV CPP "gcc-4.9 -E"
ENV CXX "g++-4.9"
ENV PATH_BEFORE_V8 "${MY_HOME}/bin:${PATH}"
ENV PATH "${MY_HOME}/bin:${PATH}"
RUN echo "deb http://ftp.us.debian.org/debian/ jessie main contrib non-free" >> /etc/apt/sources.list && \
echo "deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free" >> /etc/apt/sources.list && \
apt-get update && \
apt-get install -y gcc-4.9 g++-4.9 && \
mkdir -p ${MY_HOME}/bin && cd ${MY_HOME}/bin && \
ln -s /usr/bin/${CC} ${MY_HOME}/bin/gcc && \
ln -s /usr/bin/${CC} ${MY_HOME}/bin/x86_64-linux-gnu-gcc && \
ln -s /usr/bin/${CXX} ${MY_HOME}/bin/g++ && \
ln -s /usr/bin/${CXX} ${MY_HOME}/bin/x86_64-linux-gnu-g++
# compiling a specific version of V8 and PyV8, since older combos lead to memory leaks
RUN git clone https://github.com/muellermichel/V8_r10452.git $V8_HOME && \
git clone https://github.com/muellermichel/PyV8_r429.git $MY_LIB/pyv8 && \
cd $MY_LIB/pyv8 && python setup.py build && python setup.py install
# cleaning up
RUN PATH=${PATH_BEFORE_V8} && \
head -n -2 /etc/apt/sources.list > ${MY_HOME}/sources.list.temp && \
mv ${MY_HOME}/sources.list.temp /etc/apt/sources.list && \
apt-get update
ENV PATH "${PATH_BEFORE_V8}"
ENV CC ""
ENV CPP ""
ENV CXX ""
older version that depends on the now defunct googlecode and was made for Ubuntu 12.04:
export MY_LIB_FOLDER=[PUT-YOUR-DESIRED-INSTALL-PATH-HERE]
apt-get install -y libboost-thread-dev
apt-get install -y libboost-all-dev
apt-get install -y libboost-dev
apt-get install -y libboost-python-dev
apt-get install -y git-core autoconf libtool systemtap
apt-get install -y subversion
apt-get install -y wget
mkdir -p $MY_LIB_FOLDER/boost && cd $MY_LIB_FOLDER/boost && wget http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz && tar -xvzf boost_1_54_0.tar.gz
cd $MY_LIB_FOLDER/boost/boost_1_54_0 && ./bootstrap.sh && ./b2 install --prefix=/usr/local --with-python --with-thread && ldconfig && ldconfig /usr/local/lib
svn checkout -r10452 http://v8.googlecode.com/svn/trunk/ $MY_LIB_FOLDER/v8
export V8_HOME=$MY_LIB_FOLDER/v8
svn checkout -r429 http://pyv8.googlecode.com/svn/trunk/ $MY_LIB_FOLDER/pyv8
git clone https://github.com/taguchimail/pyv8-linux-x64.git $MY_LIB_FOLDER/pyv8-taguchimail && cd $MY_LIB_FOLDER/pyv8-taguchimail && git checkout origin/stable
apt-get install -y scons
cd $MY_LIB_FOLDER/pyv8 && patch -p0 < $MY_LIB_FOLDER/pyv8-taguchimail/patches/pyv8.patch && python setup.py build && python setup.py install

Had the same problem and this worked for me:
export LIB=~
apt-get install -y curl libboost-thread-dev libboost-all-dev libboost-dev libboost-python-dev git-core autoconf libtool
svn checkout -r19632 http://v8.googlecode.com/svn/trunk/ $LIB/v8
export V8_HOME=$LIB/v8
svn checkout http://pyv8.googlecode.com/svn/trunk/ $LIB/pyv8 && cd $LIB/pyv8 && python setup.py build && python setup.py install
Solution found in comments here - https://code.google.com/p/pyv8/wiki/HowToBuild

I'm using a Debian based distro. Here's how I installed PyV8 (you'll need to
have git installed):
cd /usr/share
sudo git clone https://github.com/emmetio/pyv8-binaries.git
cd pyv8-binaries/
sudo unzip pyv8-linux64.zip
sudo cp -a PyV8.py _PyV8.so /usr/bin