I have a requirements.txt file which contains the following package:
git+https://username:password#gitlab.mycompany.com/mypackage.git#master#egg=mypackage
I am able to build my docker image using a basic dockerfile.
However, I'm trying to use a more complex docker file to get my docker image to be as slim as possible:
FROM python:3.7-alpine as base
COPY . /app
WORKDIR /app
FROM base AS dependencies
COPY requirements.txt ./
RUN apk add --no-cache make automake gcc g++ git && \
pip install -r requirements.txt
FROM base
WORKDIR /app
COPY . /app
COPY --from=dependencies /root/.cache /root/.cache
COPY requirements.txt ./
RUN pip install -r requirements.txt && rm -rf /root/.cache
EXPOSE 8000
CMD python main.py
The problem is that during the last phase of the build I get error which 'git' cannot be found, i.e The build tries to pull 'mypackage' instead of taking it from the "dependencies" part. Any idea how to fix this?
The error:
Error [Errno 2] No such file or directory: 'git': 'git' while executing command git clone -q Cannot find command 'git' - do you have 'git' installed and in your PATH?
You don't have git in your last (3rd) image, because you only have git in dependencies, while the last one derives from base, which is pure alpine python.
So when you try to RUN pip install -r requirements.txt && rm -rf /root/.cache, you fail on requirement with git protocol.
If you need your final image to be slim, there are few options how to fix it:
use venv (Python's virtual environment); create it on 2nd step and COPY to last one. Then there no need to install requirements.
download reqs from repository to local disk on 2nd step, then COPY them to 3rd step and install (may need gcc on 3rd step, but not git)
Related
My Dockerfile
FROM python:3.7 AS builder
RUN python3 -m venv /venv
COPY requirements.txt .
RUN /venv/bin/pip3 install -r requirements.txt
FROM python:3.7
WORKDIR /home/sokov_admin/www/bot-telegram
COPY . .
CMD ["/venv/bin/python", "./bot.py"]
When I run the docker image I have this error:
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:380: starting container process caused: exec:
"/venv/bin/python": stat /venv/bin/python: no such file or directory:
unknown.
What should I change in my code?
The example you show doesn't need any OS-level dependencies for Python dependency builds. That simplifies things significantly: you can do things in a single Docker build stage, without a virtual environment, and there wouldn't be any particular benefit from splitting it up.
FROM python:3.7
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["./bot.py"]
The place where a multi-stage build with a virtual environment helps is if you need a full C toolchain to build Python libraries. In this case, in a first stage, you install the C toolchain and set up the virtual environment. In the second stage you need to COPY --from=... the entire virtual environment to the final image.
# Builder stage:
FROM python:3.7 AS builder
# Install OS-level dependencies
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
build-essential
# libmysql-client-dev, for example
# Create the virtual environment
RUN python3 -m venv /venv
ENV PATH=/venv/bin:$PATH
# Install Python dependencies
WORKDIR /app
COPY requirements.txt .
RUN pip3 install -r requirements.txt
# If your setup.py/setup.cfg has a console script entry point,
# install the application too
# COPY . .
# RUN pip3 install .
# Final stage:
FROM python:3.7 # must be _exactly_ the same image as the builder
# Install OS-level dependencies if needed (libmysqlclient, not ...-dev)
# RUN apt-get update && apt-get install ...
# Copy the virtual environment; must be _exactly_ the same path
COPY --from=builder /venv /venv
ENV PATH=/venv/bin:$PATH
# Copy in the application (if it wasn't `pip install`ed into the venv)
WORKDIR /app
COPY . .
# Say how to run it
EXPOSE 8000
CMD ["./bot.py"]
I'm trying to create a Python webapp docker image using multi-stage, to shrink the image size... right now it's around 300mb... it's also using virtual enviroment.
The docker image builds and runs fine up untill the point I need to add multi-stage so I know something is going wrong after that.... Could you help me out identifying what's wrong?
FROM python:3.8.3-alpine AS origin
RUN apk update && apk add git
RUN apk --no-cache add py3-pip build-base
RUN pip install -U pip
RUN pip install virtualenv
RUN virtualenv venv
RUN source venv/bin/activate
WORKDIR /opt/app
COPY . .
RUN pip install -r requirements.txt
## Works fine until this point ""
FROM alpine:latest
WORKDIR /opt/app
COPY --from=origin /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" VIRTUAL_ENV="/opt/venv"
COPY . /opt/app/
CMD [ "file.py" ]
ENTRYPOINT ["python"]
Without the VENV it looks something like this (still throwing error "sh: python: not found"):
FROM python:3.8.3-alpine AS origin
WORKDIR /opt/app
RUN apk update && apk add git
RUN apk --no-cache add py3-pip build-base
RUN pip install -U pip
COPY . .
RUN pip install -r requirements.txt
FROM alpine:latest
WORKDIR /home
COPY --from=origin /opt/app .
CMD sh -c 'python file.py'
You still need pyhton in your runtime container, since you changed your last image to just alpine it wouldn't work. Just a tip, combine your CMD and ENTRYPOINT under one of them, there is generally no need for having two of them. Try to use only ENTRYPOINT since you can pass CMD easily in runtime for example to activate debug mode more easily.
EDIT: Please stay away from alpine for python apps as you can get some weird issues about it. You can use "python_version-slim-buster" images, they are small enough.
In my docker django project i need for read/write purpose to create a volumes in my Dockerile and install/run app on it.
i found this article : DockerFile on StackOverflow but sincerly i don't understand more about it.
Here my Dockerfile:
FROM python:3.6-alpine
EXPOSE 8000
RUN apk update
RUN apk add --no-cache make linux-headers libffi-dev jpeg-dev zlib-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN mkdir /Code
VOLUME /var/lib/cathstudio/data
WORKDIR /Code
COPY ./requirements.txt .
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
ENV PYTHONUNBUFFERED 1
COPY . /Code/
ENTRYPOINT python /Code/core/manage.py runserver 0.0.0.0:8000
at my original file i add the VOLUME /var/lib/cathstudio/data instruction, but after that how can i say to the rest of my code to use that volumes for WORKDIR, install requirements.txt, copy code and run app?
i don't what to specify it in RUN statement with -v directive after build, i would integrate the volume creation and manage directly in dockerfile.
So many thanks in advance
for anything expect pip you may specify workdir once:
WORKDIR /var/lib/cathstudio/data
for pip use -t or --target:
pip install -t /var/lib/cathstudio/data
-t, --target
Install packages into <dir>. By default this will not replace existing files/folders in <dir>. Use --upgrade to replace existing
packages in with new versions
I would to create a docker image from my django app but I don't want to pull my .py file.
For doing this I need a way to compile my python file in docker then remove all *.py and left only *.pyc file.
If I simply write **/*.py in the .dokerignore file, I get the error
ImportError: bad magic number in 'ajaxfuncs': b'\x03\xf3\r\n'
because my original *.py file was compiled with a different python version (local python3.6.0 docker python3.6 for alpine)
So as workaround I would build my python file in dockerfile and then remove all py
First in my .dockerignorefile i put: **/*.pyc
Then in my Dockerfile think to use python -m py_compile for generate my new *.pyc files:
FROM python:3.6-alpine
EXPOSE 8000
RUN apk update
RUN apk add --no-cache make linux-headers libffi-dev jpeg-dev zlib-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN mkdir /Code
WORKDIR /Code
COPY ./requirements.txt .
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
ENV PYTHONUNBUFFERED 1
COPY . /Code/
RUN python -m py_compile /Code/ajaxfuncs/ajax.py /Code/ajaxfuncs/asktempl.py /Code/ajaxfuncs/group.py /Code/ajaxfuncs/history.py /Code/ajaxfuncs/template_import.py
RUN rm -rf /Code/ajaxfuncs/*.py
ENTRYPOINT python /Code/core/manage.py runserver 0.0.0.0:8000
but when I run my application seem that compiler does not compile my files, because no pyc was found.
If I remove the pyc entry in .dockerignore I again get the error:
ImportError: bad magic number in 'ajaxfuncs': b'\x03\xf3\r\n'
Does someone know how I can compile python file during docker container creation or another method for avoid *.py file in container?
So many thanks in advance
I have a requirements.txt file containing, amongst others:
Flask-RQ==0.2
-e git+https://token:x-oauth-basic#github.com/user/repo.git#egg=repo
When I try to build a Docker container using Docker Compose, it downloads both packages, and install them both, but when I do a pip freeze there is no sign of the -e package. When I try to run the app, it looks as if this package hasn't been installed. Here's the relevant output from the build:
Collecting Flask-RQ==0.2 (from -r requirements.txt (line 3))
Downloading Flask-RQ-0.2.tar.gz
Obtaining repo from git+https://token:x-oauth-basic#github.com/user/repo.git#egg=repo (from -r requirements.txt (line 4))
Cloning https://token:x-oauth-basic#github.com/user/repo.git to ./src/repo
And here's my Dockerfile:
FROM python:2.7
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN pip install -r requirements.txt
COPY . /usr/src/app
I find this situation very strange and would appreciate any help.
I ran into a similar issue, and one possible way that the problem can appear is from:
WORKDIR /usr/src/app
being set before pip install. pip will create the src/ directory (where the package is installed) inside of the WORKDIR. Now all of this shouldn't be an issue since your app files, when copied over, should not overwrite the src/ directory.
However, you might be mounting a volume to /usr/src/app. When you do that, you'll overwrite the /usr/src/app/src directory and then your package will not be found.
So one fix is to move WORKDIR after the pip install. So your Dockerfile will look like:
FROM python:2.7
RUN mkdir -p /usr/src/app
COPY requirements.txt /usr/src/app/
RUN pip install -r /usr/src/app/requirements.txt
COPY . /usr/src/app
WORKDIR /usr/src/app
This fixed it for me. Hopefully it'll work for you.
#mikexstudios is correct, this happens because pip stores the package source in /usr/src/app/src, but you're mounting a local directory over top of it, meaning python can't find the package source.
Rather than changing the position of WORKDIR, I solved it by changing the pip command to:
pip install -r requirements.txt --src /usr/local/src
Either approach should work.
If you are recieving a similar error when installing a git repo from a requirements file under a dockerized container, you may have forgotten to install git.
Here is the error I recieved:
Downloading/unpacking CMRESHandler from
git+git://github.com/zigius/python-elasticsearch-logger.git (from -r
/home/ubuntu/requirements.txt (line 5))
Cloning git://github.com/zigius/python-elasticsearch-logger.git to
/tmp/pip_build_root/CMRESHandler
Cleaning up...
Cannot find command 'git'
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c useradd ubuntu -b /home && echo
"ubuntu ALL = NOPASSWD: ALL" >> /etc/sudoers &&
chown -R ubuntu:ubuntu /home/ubuntu && pip install -r /home/ubuntu/requirements.txt returned a non-zero code: 1
Here is an example Dockerfile that installs git and then installs all requirements:
FROM python:3.5-slim
RUN apt-get update && apt-get install -y --no-install-recommends git \
ADD . /code
WORKDIR /code
RUN pip install --upgrade pip setuptools && pip install -r /home/ubuntu/requirements.txt
Now you can use git packages in your requirements file in a Dockerized environment