I need both java and python in my docker container to run some code.
This is my dockerfile:
It works perpectly if I don't add the FROM openjdk:slim
#get python
FROM python:3.6-slim
RUN pip install --trusted-host pypi.python.org flask
#get openjdk
FROM openjdk:slim
COPY . /targetdir
WORKDIR /targetdir
# Make port 81 available to the world outside this container
EXPOSE 81
CMD ["python", "test.py"]
And the test.py app is in the same directory:
from flask import Flask
import os
app = Flask(__name__)
#app.route("/")
def hello():
html = "<h3>Test:{test}</h3>"
test = os.environ['JAVA_HOME']
return html.format(test = test)
if __name__ == '__main__':
app.run(debug=True,host='0.0.0.0',port=81)
I'm getting this error:
D:\MyApps\Docker Toolbox\Docker Toolbox\docker.exe: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"python\": executable file not found in $PATH": unknown.
What exactly am I doing wrong here? I'm new to docker, perhaps I'm missing a step.
Additional details
My goal
I have to run a python program that runs a Java file. The python library I'm using requires the path to JAVA_HOME.
My issues:
I do not know Java, so I cannot run the file properly.
My entire code is in Python, except this Java bit
The Python wrapper runs the file in a way I need it to run.
An easier solution to the above issue is to use multi-stage docker containers where you can copy the content from one to another. In the above case you can have openjdk:slim as the base container and then use content from a python container to be copied over into this base container as follows:
FROM openjdk:slim
COPY --from=python:3.6 / /
...
<normal instructions for python container continues>
...
This feature is available as of Docker 17.05 and there are more things you can do using multi-stage build as in copying only the content you need from one to another.
Reference documentation
OK it took me a little while to figure it out. And my thanks go to this answer.
I think my approach didn't work because I did not have a basic version of Linux.
So it goes like this:
Get Linux (I'm using Alpine because it's barebones)
Get Java via the package manager
Get Python, PIP
OPTIONAL: find and set JAVA_HOME
Find the path to JAVA_HOME. Perhaps there is a better way to do this, but I did this running the running the container, then I looked inside the container using docker exec -it [COINTAINER ID] bin/bash and found it.
Set JAVA_HOME in dockerfile and build + run it all again
Here is the final Dockerfile ( it should work with the python code in the question) :
### 1. Get Linux
FROM alpine:3.7
### 2. Get Java via the package manager
RUN apk update \
&& apk upgrade \
&& apk add --no-cache bash \
&& apk add --no-cache --virtual=build-dependencies unzip \
&& apk add --no-cache curl \
&& apk add --no-cache openjdk8-jre
### 3. Get Python, PIP
RUN apk add --no-cache python3 \
&& python3 -m ensurepip \
&& pip3 install --upgrade pip setuptools \
&& rm -r /usr/lib/python*/ensurepip && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache
### Get Flask for the app
RUN pip install --trusted-host pypi.python.org flask
####
#### OPTIONAL : 4. SET JAVA_HOME environment variable, uncomment the line below if you need it
#ENV JAVA_HOME="/usr/lib/jvm/java-1.8-openjdk"
####
EXPOSE 81
ADD test.py /
CMD ["python", "test.py"]
I'm new to Docker, so this may not be the best possible solution. I'm open to suggestions.
UPDATE: COMMON ISUUES
Difficulty using python packages
As Joabe Lucena pointed out here, Alpine can have issues certain python packages.
I recommend that you use a Linux distro that works best for you, e.g. centos.
Another alternative is to simply use docker-java-python image from docker hub. https://hub.docker.com/r/rappdw/docker-java-python
FROM rappdw/docker-java-python:openjdk1.8.0_171-python3.6.6
RUN java -version
RUN python --version
I found Sunny Pal's answer very useful but I made the copy more specific and added the necessary environment variables and update-alternatives lines so that Java was accessible from the command line in the Python container.
FROM python:3.9-slim
COPY --from=openjdk:8-jre-slim /usr/local/openjdk-8 /usr/local/openjdk-8
ENV JAVA_HOME /usr/local/openjdk-8
RUN update-alternatives --install /usr/bin/java java /usr/local/openjdk-8/bin/java 1
...
Oh, let me add my five cents. I took python slim as a base image. Then I found open-jdk-11 (Note, open-jdk-10 will fail because it is not supported) base image code!... And copy-pasted it into my docker file.
Note, copy-paste driven development is cool... ONLY when you understand each line you use in your code!!!
And here it is!
<!-- language: shell -->
FROM python:3.7.2-slim
# Do your stuff, install python.
# and now Jdk
RUN rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update && apt-get upgrade -y \
&& apt-get install -y --no-install-recommends curl ca-certificates \
&& rm -rf /var/lib/apt/lists/*
ENV JAVA_VERSION jdk-11.0.2+7
COPY slim-java* /usr/local/bin/
RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
ppc64el|ppc64le) \
ESUM='c18364a778b1b990e8e62d094377af48b000f9f6a64ec21baff6a032af06386d'; \
BINARY_URL='https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.1%2B13/OpenJDK11U-jdk_ppc64le_linux_hotspot_11.0.1_13.tar.gz'; \
;; \
s390x) \
ESUM='e39aacc270731dadcdc000aaaf709adae7a08113ccf5b4a045bc87fc13458d71'; \
BINARY_URL='https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11%2B28/OpenJDK11-jdk_s390x_linux_hotspot_11_28.tar.gz'; \
;; \
amd64|x86_64) \
ESUM='d89304a971e5186e80b6a48a9415e49583b7a5a9315ba5552d373be7782fc528'; \
BINARY_URL='https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.2%2B7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.2_7.tar.gz'; \
;; \
aarch64|arm64) \
ESUM='b66121b9a0c2e7176373e670a499b9d55344bcb326f67140ad6d0dc24d13d3e2'; \
BINARY_URL='https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.1%2B13/OpenJDK11U-jdk_aarch64_linux_hotspot_11.0.1_13.tar.gz'; \
;; \
*) \
echo "Unsupported arch: ${ARCH}"; \
exit 1; \
;; \
esac; \
curl -Lso /tmp/openjdk.tar.gz ${BINARY_URL}; \
sha256sum /tmp/openjdk.tar.gz; \
mkdir -p /opt/java/openjdk; \
cd /opt/java/openjdk; \
echo "${ESUM} /tmp/openjdk.tar.gz" | sha256sum -c -; \
tar -xf /tmp/openjdk.tar.gz; \
jdir=$(dirname $(dirname $(find /opt/java/openjdk -name javac))); \
mv ${jdir}/* /opt/java/openjdk; \
export PATH="/opt/java/openjdk/bin:$PATH"; \
apt-get update; apt-get install -y --no-install-recommends binutils; \
/usr/local/bin/slim-java.sh /opt/java/openjdk; \
apt-get remove -y binutils; \
rm -rf /var/lib/apt/lists/*; \
rm -rf ${jdir} /tmp/openjdk.tar.gz;
ENV JAVA_HOME=/opt/java/openjdk \
PATH="/opt/java/openjdk/bin:$PATH"
ENV JAVA_TOOL_OPTIONS="-XX:+UseContainerSupport"
Now references.
https://github.com/AdoptOpenJDK/openjdk-docker/blob/master/11/jdk/ubuntu/Dockerfile.hotspot.releases.slim
https://hub.docker.com/_/python/
https://hub.docker.com/r/adoptopenjdk/openjdk11/
I used them to answer this question, which may help you sometime.
Running Python and Java in Docker
I believe that by adding FROM openjdk:slim line, you tell docker to execute all of your subsequent commands in openjdk container (which does not have python)
I would approach this by creating two separate containers for openjdk and python and specify individual sets of commands for them.
Docker is made to modularize your solutions and mashing everything into one container is usually a bad practice.
I tried pajamas's anwser which worked very well for creating this image. However, when trying to install packages like gensim, pandas or else, I faced some errors like: don't know how to compile Fortran code on platform 'posix'. I searched and tried this, this and that but none worked for me.
So, based on pajamas's anwser I decided to convert his image from Alpine to Centos which worked very well. So here's a Dockerfile that might help someone who's may be struggling in this scenario like I was:
# Get Linux
FROM centos:7
# Install Java
RUN yum update -y \
&& yum install java-1.8.0-openjdk -y \
&& yum clean all \
&& rm -rf /var/cache/yum
# Set JAVA_HOME environment var
ENV JAVA_HOME="/usr/lib/jvm/jre-openjdk"
# Install Python
RUN yum install python3 -y \
&& pip3 install --upgrade pip setuptools wheel \
&& if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi \
&& if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi \
&& yum clean all \
&& rm -rf /var/cache/yum
CMD ["bash"]
you should have one FROM in your dockerfile
(unless you use multi-stage build for the docker)
I think i found easiest way to mix java jdk 17 and python3. I is not working on python2
FROM openjdk:17.0.1-jdk-slim
RUN apt-get update && \
apt-get install -y software-properties-common && \
apt-get install -y python3-pip
Software Commons have python3 lightweight version. (3.9.1 version)
U can also install some libraries like that.
RUN python3 -m pip install --upgrade pip && \
python3 -m pip install numpy && \
python3 -m pip install opencv-python
OR
RUN apt-get update && \
apt-get install -y ffmpeg
Easiest is to just start from a Python image and add the OpenJDK. Note that FROM openjdk has been deprecated and replaced with eclipse-temurin
FROM python:3.10
ENV JAVA_HOME=/opt/java/openjdk
COPY --from=eclipse-temurin:17-jre $JAVA_HOME $JAVA_HOME
ENV PATH="${JAVA_HOME}/bin:${PATH}"
RUN pip install --trusted-host pypi.python.org flask
See How to use this Image - Using a different base Image section of https://hub.docker.com/_/eclipse-temurin for details.
Instead of using FROM openjdk:slim you can separately install Java, please refer below example:
# Install OpenJDK-8
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get clean;
# Fix certificate issues
RUN apt-get update && \
apt-get install ca-certificates-java && \
apt-get clean && \
update-ca-certificates -f;
# Setup JAVA_HOME -- useful for docker commandline
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME
A docker image I am creating and sending to a client is somehow deleting its source code 24-48 hours after it is started. We can see this by exec onto the running container and talking a look around.
The service is a simple flask app. The service doesn't go down as the application doesn't experience an issue but the static files it should be yielding go missing (along with everything else copied in) so we start getting 404s. I can't think of anything that would explain this (especially considering that it takes time for it to occur)
FROM python:3.8-slim-buster
ARG USERNAME=calibrator
ARG USER_UID=1000
ARG USER_GID=$USER_UID
RUN apt-get update \
&& groupadd --gid $USER_GID $USERNAME \
&& useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME \
&& apt-get install -y sudo \
&& echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME\
&& chmod 0440 /etc/sudoers.d/$USERNAME \
# Install open-cv packaged
&& apt-get install -y libsm6 libxext6 libxrender-dev libgtk2.0-dev libgl1-mesa-glx \
#
## Git
&& sudo apt-get install -y git-lfs \
#
## Bespoke setup
&& apt-get -y install unixodbc-dev \
#
## PostgresSQL
&& apt-get -y install libpq-dev
ENV PATH="/home/${USERNAME}/.local/bin:${PATH}"
ARG git_user
ARG git_password
RUN pip install --upgrade pip
RUN python3 -m pip install --user git+https://${git_user}:${git_password}#bitbucket.org/****
WORKDIR /home/calibrator
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY app app
ENV FLASK_APP=app/app.py
EXPOSE 80
STOPSIGNAL SIGTERM
CMD ["uwsgi", "--http", ":80", "--module", "app.app", "--callable", "app", "--processes=1", "--master"]
version: "3.7"
services:
calibrator:
container_name: sed-calibrator-ui
image: sed-metadata-calibrator:2.0.3
restart: always
ports:
- "8081:80"
environment:
- STORE_ID=N0001
- DYNAMO_TABLE=****
- DYNAMO_REGION=****
- AWS_DEFAULT_REGION=****
- AWS_ACCESS_KEY_ID=****
- AWS_SECRET_ACCESS_KEY=****
The application reads in a single configuration file and connects to a database on startup and then defines the endpoints - none of which touch the filesystem again. How can the source code be deleting itself!?
Creating a new container resolves the issue.
Any suggestions in checking the client's environment would be appreciated because I cannot replicate it.
Clients versions
Docker Version - 18.09.7
Docker Compose version - 1.24.0
I was able to solve the problem by updating the kernel, it also worked with an older kernel (3.10)
works:
4.1.12-124.45.6.el7uek.x86_64
not work:
4.1.12-124.43.4.el7uek.x86_64
I do not know the reason that causes it, I only know that after updating the kernel the problem was solved. I hope it is your same problem
I have an application that consists of a docker container with a redis instance, another docker container with a rabbitmq server, and a third container in which I'm trying to start a number of celery workers that can interact with the redis and rabbitmq containers. I'm starting and stopping these workers via a rest api, and have checked that I'm able to do this on my host machine. However, after moving the setup to docker, it seems the workers are not behaving as expected. Whereas on my host machine (windows 10) I was able to see the reply from the rest api and console output from the workers, I can only see the response from the rest api (a log message) and no console output. It also seems that the workers are not accessing the redis and rabbitmq instances.
My docker container is built from a python3.6 (linux) base image. I have checked that everything is installed correctly, and there are no error logs. I build the image with the following dockerfile:
FROM python:3.6
WORKDIR /opt
# create a virtual environment and add it to PATH so that it is applied for
# all future RUN and CMD calls
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# install msodbcsql17
RUN apt-get update \
&& apt-get install -y curl apt-transport-https gnupg2 \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
\
&& curl https://packages.microsoft.com/config/debian/9/prod.list >
/etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql17 mssql-tools
# Install Mono for pythonnet.
RUN apt-get update \
&& apt-get install --yes \
dirmngr \
clang \
gnupg \
ca-certificates \
# Dependency for pyodbc.
unixodbc-dev \
&& apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys
3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF \
&& echo "deb http://download.mono-project.com/repo/debian
stretch/snapshots/5.20 main" | tee /etc/apt/sources.list.d/mono-official-
stable.list \
&& apt-get update \
&& apt-get install --yes \
mono-devel=5.20\* \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
COPY src ./src
COPY setup.py ./setup.py
COPY config.json ./config.json
COPY Utility.dll ./Utility.dll
COPY settings_docker.ini ./settings.ini
COPY config.json ./config.json
COPY sql_config.json ./sql_config.json
RUN python3 -m venv $VIRTUAL_ENV \
# From here on, use virtual env's python.
&& venv/bin/pip install --upgrade pip \
&& venv/bin/pip install --no-cache-dir --upgrade pip setuptools wheel \
&& venv/bin/pip install --no-cache-dir -r requirements.txt \
# Dependency for pythonnet.
&& venv/bin/pip install --no-cache-dir pycparser \
&& venv/bin/pip install -U --no-cache-dir "pythonnet==2.5.1" \
# && python -m pip install --no-cache-dir "pyodbc==4.0.25" "pythonnet==2.5.1"
EXPOSE 8081
cmd python src/celery_artifacts/docker_workers/worker_app.py
And then run it with this command:
docker run --name app -p 8081:8081 app
I then attach the container to the same bridge network as the other 2:
docker network connect my_network app
Is there a way to see the same console output from my container as the one on the host?
I have a Dockerfile that fails on build with the error;
COPY failed: stat /var/lib/docker/tmp/docker-builder476469130/requirements.txt: no such file or directory
The error occurs on the COPY line for the requirments.txt file. I use a pretty standard Dockerfile;
FROM python:3.6.7-slim
# Version: 1.4
# Dockerfile to build the coroner container.
# Install Python and Package Libraries
RUN apt-get update && apt-get upgrade -y && apt-get autoremove && apt-get autoclean
RUN apt-get install -y \
libffi-dev \
libssl-dev \
default-libmysqlclient-dev \
libxml2-dev \
libxslt-dev \
libjpeg-dev \
libfreetype6-dev \
zlib1g-dev \
net-tools \
nano
ARG PROJECT=coroner
ARG PROJECT_DIR=/var/www/${PROJECT}
WORKDIR $PROJECT_DIR
ENV PYTHONUNBUFFERED 1
RUN mkdir -p $PROJECT_DIR
COPY requirements.txt $PROJECT_DIR/requirments.txt
RUN pip install --upgrade pip
RUN pip install -r $PROJECT_DIR/requirements.txt
EXPOSE 8888
STOPSIGNAL SIGINT
ENTRYPOINT ["python", "manage.py"]
CMD ["runserver", "0.0.0.0:8888"]
I am bashing my head against this and have been praying at the church of google for a while now. I have checked the context and it seems to be correct. my build command is;
sudo docker build -t coroner:dev .
Docker Version Docker version 19.03.6, build 369ce74a3c
Can somebody put me out of my misery, please?
You've got a typo in 'requirements.txt' in the destination, you've put 'requirments.txt'.
However, because you're simply copying this to where you've specified your WORKDIR, you can just do:
COPY requirements.txt .
The file will then be copied into your CWD.
I have posted this question over at docker hub as well as I am not sure which community would have more of a response to my question. I am new to creating docker images and have put together a Dockerfile that creates an image that works but the final image size is 600MB+ and would like somebody that is more advance than I to advise if there is anything I can do to reduce this. I have read many blogs about various strategies to do so and have gone the Python virtual environment route. I am really concerned about build times as I will not be building often but would like to see the final image size be a bit leaner than what it is.
What I am doing is building an image with a python application, fava which is a web gui front-end to the accounting program beancount. These two python applications alone are easy enough and the fava team even provide a Dockerfile based on alpine to build a light image. However, the issue I have is I want to extend this by including an extension to this python application (smart_importer) which provides some machine learning features that will automate aspects of the transaction importing process. This extension depends on numpy, scipy, and scikit-learn and this is where the extra weight comes from. I originally tried to extended the alpine approach that the fava team had but installing scipy on alpine fails horribly which I cannot resolve. Using python slim I can build a fairly small (<200mb) final image with just fava and beancount...but as I said this balloons with the introduction of the needed dependencies of smart_importer.
Here is the Dockerfile I have currently, are there any changes I can easily make to get the final image size down that I am not seeing? Greatly appreciate any pointers one might have.
FROM python:slim AS base
FROM base as builder
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
#WORKDIR /install
COPY ./library-dependencies.txt /tmp/library-dependencies.txt
COPY ./requirements.txt /tmp/requirements.txt
#ENV PATH="/install:${PATH}"
RUN buildDeps='build-essential gcc gfortran python3-dev' \
&& apt-get update \
&& apt-get install -y $buildDeps --no-install-recommends \
&& cat /tmp/library-dependencies.txt | egrep -v "^\s*(#|$)" | xargs apt-get install -y \
&& pip3 install --upgrade pip \
&& CFLAGS="-g0 -Wl,--strip-all -I/usr/include:/usr/local/include -L/usr/lib:/usr/local/lib" \
pip3 install \
# --prefix="/install" \
--no-cache-dir \
--compile \
--global-option=build_ext \
--global-option="-j 6" \
-r /tmp/requirements.txt \
&& apt-get purge -y --auto-remove $buildDeps \
&& rm -rf /var/lib/apt/lists/* \
&& rm -r \
/tmp/requirements.txt \
/tmp/library-dependencies.txt
FROM base
COPY --from=builder /opt/venv /opt/venv
COPY ./library-dependencies.txt /tmp/library-dependencies.txt
RUN apt-get update \
&& cat /tmp/library-dependencies.txt | egrep -v "^\s*(#|$)" | xargs apt-get install -y \
&& apt-get install -y libgomp1 --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
ENV PATH="/opt/venv/bin:$PATH"
ENV BEANCOUNT_FILE ""
ENV FAVA_OPTIONS ""
EXPOSE 5000
CMD fava --host 0.0.0.0 $FAVA_OPTIONS $BEANCOUNT_FILE
requirements.txt
# numeric packages needed for smart_importer
Cython==0.28.5
numpy==1.15.1
scipy==1.1.0
scikit-learn
#fava
fava
smart_importer
library-dependencies.txt
libopenblas-dev
liblapack-dev
libxml2-dev
libxslt1-dev
zlib1g-dev
Does installing the py3-scipy package with apk work?
I would recommend Python on Alpine Linux. Any Dockerfile changes you make are not going to get your image nearly as small as Alpine.
This looks like a good start: https://github.com/jfloff/alpine-python