I want to install Python, Pip and virtualenv in a Docker container which is Ubuntu, I create the image through Dockerfile:
FROM ubuntu:16.04
RUN apt-get update -y
RUN apt-get
RUN apt-get python3 -y
RUN apt-get install python3-pip -y
RUN pip install virtualenv
...
When it went here, it came out "/bin/sh: 1: pip: not found", but there's no exception showed in the installation process, is this mean I didn't install pip right? Or should I do anything else before I use pip order?
Then I changed the Dockerfile like this:
...
RUN apt-get python3 -y
RUN apt-get install python3-pip -y
RUN apt-get install python3-virtualenv -y
RUN virtualenv --no-stie-packages -p /path/python3 py3env
...
but it still saying /bin/sh: 1: virtualenv: not found, here is it
I also installed git, git clone order ran correctly, can be used. Where am I wrong, or how should I do?
On Debian-based platforms, including Ubuntu, the command installed by python3-pip is called pip3 in order for it to peacefully coexist with any system-installed Python 2 and its pip.
Somewhat similarly, the virtualenv command is not installed by the package python3-virtualenv; to get that, you need apt-get install -y virtualenv.
Note that venv is included in the Python 3 standard library, so you don't really need to install anything at all.
python3 -m venv newenv
Why would you want a virtualenv inside Docker anyway, though? (There are situations where it makes sense but in the vast majority of cases, you want the Docker container to be as simple as possible, which means, install everything as root, and rebuild the whole container if something needs to be updated.)
As an aside, you generally want to minimize the number of RUN statements. Making many layers while debugging is perhaps defensible, but layers which do nothing are definitely just wasteful. Perhaps also discover that apt-get can install more than one package at a time.
RUN apt-get update -y && \
apt-get install -y python3 python3-pip && \
...
The && causes the entire RUN sequence to fail as soon as one of the commands fails.
What sense to use virtual environment inside container? Virtualenv very helpful for local development, it allows you to use different versions of python interpreter and packages on a single machine. But in your docker container should be only one process (actually container it is process), and you can install all your requirements globally.
But if you really have strong reasons for that, you probably must use "python3 way" to create virtual env.
So your Dockerfile should look something like that:
FROM ubuntu:16.04
RUN apt-get update -y \
&& apt install python3 -y \
&& apt install python3-pip -y \
&& apt install python3-venv -y \
&& python3 -m venv venv
ENTRYPOINT bin/bash
You can build it with command
docker build -t ubuntu-python .
And run with
docker run --rm -it ubuntu-python
In container shell you could activate venv with command
source venv/bin/activate
then run python interpreter and check that it was run from venv:
>>> import sys
>>> sys.executable
it should print /venv/bin/python
I have no idea how to run container with preactivated (I don't know if this word really exists) virtual environment and I still think you actually don't need to use virtual environment in your container.
Also you'd better try ready python-images for docker, e.g. light alpine images instead of extending basic ubuntu image.
Excuse my terrible russian-english, I hope you'll understand my answer :)
Related
I want the exact version of Python 3.8.13 to run on my container, but so far when I used the below, it generated a very large Docker image:
RUN yum update -y && yum install -y python3.8 python38-pip && yum clean all
The command "yum install python3.8" installs 3.8.13 and this is fine, but as mentioned, the end result (with other required elements) is a bit above 2 GB once built. I would like to make the image smaller and I am wondering if I can use a slim or alpine version of Python 3.8.13.
I was trying with the following commands:
yum install -y python3.8.13-slim
yum install -y python3.8.13-slim-buster
yum install -y python3.8-slim
Did not succeed, yum does not recognize these as valid packages.
Is there a workaround for this?
Some background : I'm new to understanding docker images and containers and how to write DOCKERFILE. I currently have a Dockerfile which installs all the dependencies that I want through PIP install command and so, it was very simple to build and deploy images.
But I currently have a new requirement to use the Dateinfer module and that cannot be installed through the pip install command.
The repo has to be first cloned and then has to be installed and I'm having difficulty achieving this through a DOCKERFILE. The current work around I've been following for now is to run the container and install it manually in the directory with all the other dependencies and Committing the changes with dateinfer installed.But this is a very tedious and time consuming process and I want to achieve the same by just mentioning it in the DOCKERFILE along with all my other dependencies.
This is what my Dockerfile looks like:
FROM ubuntu:20.04
RUN apt update
RUN apt upgrade -y
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
RUN apt-get install -y libenchant1c2a
RUN apt install git -y
RUN pip3 install argparse
RUN pip3 install boto3
RUN pip3 install numpy==1.19.1
RUN pip3 install scipy
RUN pip3 install pandas
RUN pip3 install scikit-learn
RUN pip3 install matplotlib
RUN pip3 install plotly
RUN pip3 install kaleido
RUN pip3 install fpdf
RUN pip3 install regex
RUN pip3 install pyenchant
RUN pip3 install openpyxl
ADD core.py /
ENTRYPOINT [ "/usr/bin/python3.8", "/core.py”]
So when I try to install Dateinfer like this:
RUN git clone https://github.com/nedap/dateinfer.git
RUN cd dateinfer
RUN pip3 install .
It throws the following error :
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
The command '/bin/sh -c pip3 install .' returned a non-zero code: 1
How do I solve this?
Each RUN directive in a Dockerfile runs in its own subshell. If you write something like this:
RUN cd dateinfer
That is a no-op: it starts a new shell, changes directory, and then the shell exits. When the next RUN command executes, you're back in the / directory.
The easiest way of resolving this is to include your commands in a single RUN statement:
RUN git clone https://github.com/nedap/dateinfer.git && \
cd dateinfer && \
pip3 install .
In fact, you would benefit from doing this with your other pip install commands as well; rather than a bunch of individual RUN
commands, consider instead:
RUN pip3 install \
argparse \
boto3 \
numpy==1.19.1 \
scipy \
pandas \
scikit-learn \
matplotlib \
plotly \
kaleido \
fpdf \
regex \
pyenchant \
openpyxl
That will generally be faster because pip only needs to resolve
dependencies once.
Rather than specifying all the packages individually on the command
line, you could also put them into a requirements.txt file, and then
use pip install -r requirements.txt.
I have installed a library called fastai==1.0.59 via requirements.txt file inside my Dockerfile.
But the purpose of running the Django app is not achieved because of one error. To solve that error, I need to manually edit the files /site-packages/fastai/torch_core.py and site-packages/fastai/basic_train.py inside this library folder which I don't intend to.
Therefore I'm trying to copy the fastai folder itself from my host machine to the location inside docker image.
source location: /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/
destination location: ../venv/lib/python3.6/site-packages/ which is inside my docker image.
being new to docker, I tried this using COPY command like:
COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/
which gave me an error:
ERROR: Service 'app' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder583041406/Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai: no such file or directory.
I tried referring this: How to include files outside of Docker's build context?
but seems like it bounced off my head a bit..
Please help me tackling this. Thanks.
Dockerfile:
FROM python:3.6-slim-buster AS build
MAINTAINER model1
ENV PYTHONUNBUFFERED 1
RUN python3 -m venv /venv
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y git && \
apt-get install -y build-essential && \
apt-get install -y awscli && \
apt-get install -y unzip && \
apt-get install -y nano && \
apt-get install -y libsm6 libxext6 libxrender-dev
RUN apt-cache search mysql-server
RUN apt-cache search libmysqlclient-dev
RUN apt-get install -y libpq-dev
RUN apt-get install -y postgresql
RUN apt-cache search postgresql-server-dev-9.5
RUN apt-get install -y libglib2.0-0
RUN mkdir -p /model/
COPY . /model/
WORKDIR /model/
RUN pip install --upgrade awscli==1.14.5 s3cmd==2.0.1 python-magic
RUN pip install -r ./requirements.txt
EXPOSE 8001
RUN chmod -R 777 /model/
COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/
CMD python3 -m /venv/activate
CMD /model/my_setup.sh development
CMD export API_ENV = development
CMD cd server && \
python manage.py migrate && \
python manage.py runserver 0.0.0.0:8001
Short Answer
No
Long Answer
When you run docker build the current directory and all of its contents (subdirectories and all) are copied into a staging area called the 'build context'. When you issue a COPY instruction in the Dockerfile, docker will copy from the staging area into a layer in the image's filesystem.
As you can see, this procludes copying files from directories outside the build context.
Workaround
Either download the files you want from their golden-source directly into the image during the build process (this is why you often see a lot of curl statements in Dockerfiles), or you can copy the files (dirs) you need into the build-tree and check them into source control as part of your project. Which method you choose is entirely dependent on the nature of your project and the files you need.
Notes
There are other workarounds documented for this, all of them without exception break the intent of 'portability' of your build. The only quality solutions are those documented here (though I'm happy to add to this list if I've missed any that preserve portability).
I have javascript application, in which I use python utility (canconvert of canmatrix utility).
In javascript I call canconvert through
execSync(`canconvert\\
--jsonExportAll\\
--jsonNativeTypes\\
--additionalFrameAttributes\\
--additionalSignalAttributes\\
${dbcFileLocation} ${parsedJsonLocation}`);
so, canconvert should be available in the docker environment.
for now, I just install canmatrix through pip but size of the container becomes very big.
FROM node:10.15.3-slim AS dist
...
RUN apt-get update && \
apt-get install -y python python-pip && \
pip install canmatrix && \
chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
Python is not tiny. Batteries included.
I guess you could try building your python code with nuitka or cython, but then you might end up giving up on automatic upgrades.
Disk is cheap - why not just use it like you have it?
I am using a centos os base image and installing python3 with the following dockerfile
FROM centos:7
ENV container docker
ARG USER=dsadmin
ARG HOMEDIR=/home/${USER}
RUN yum clean all \
&& yum update -q -y -t \
&& yum install file -q -y
RUN useradd -s /bin/bash -d ${HOMEDIR} ${USER}
RUN export LC_ALL=en_US.UTF-8
# install Development Tools to get gcc
RUN yum groupinstall -y "Development Tools"
# install python development so that pip can compile packages
RUN yum -y install epel-release && yum clean all \
&& yum install -y python34-setuptools \
&& yum install -y python34-devel
# install pip
RUN easy_install-3.4 pip
# install virtualenv or virtualenvwrapper
RUN pip3 install virtualenv \
&& pip3 install virtualenvwrapper \
&& pip3 install pandas
# # install django
# RUN pip3 install django
USER ${USER}
WORKDIR ${HOMEDIR}
I build and tag the above as follows:
docker build . --label validation --tag validation
I then need to add a .tar.gz file to the home directory. This file contains all the python scripts I maintain. This file will change frequently. If I add it to the dockerfile above, python is installed every time I change the .gz file. This adds a lot of time to development. As a workaround, I tried creating a second dockerfile file that uses the above image as the base and then just adds the .tar.gz file on it.
FROM validation:latest
ARG USER=dsadmin
ARG HOMEDIR=/home/${USER}
ADD code/validation_utility.tar.gz ${HOMEDIR}/.
USER ${USER}
WORKDIR ${HOMEDIR}
After that if I run docker image and do an ls, all the files in the folder have a owner of games.
-rw-r--r-- 1 501 games 35785 Nov 2 21:24 Validation_utility.py
To fix the above, I added the following lines to the second docker file:
ADD code/validation_utility.tar.gz ${HOMEDIR}/.
RUN chown -R ${USER}:${USER} ${HOMEDIR} \
&& chmod +x ${HOMEDIR}/Validation_utility.py
but I get the error:
chown: changing ownership of '/home/dsadmin/Validation_utility.py': Operation not permitted
The goal is to have two docker files. The users will run the first docker file to install centos and python dependencies. The second dockerfile will install the custom python scripts. If the scripts change, they will just run the second docker file again. Is that the right way to think about docker? Thank you.
Is that the right way to think about docker?
This is the easy part of your question. Yes. You're thinking about the proper way to structure your Dockerfiles, reuse them, and keep your image builds efficient. Good job.
As for the error you're receiving, I'm less confident in answering why the ADD command is un-tarballing your tar.gz as the games user. I'm not nearly as familiar with CentOS. That's the start of the problem. dsadmin, as a regular non-privileged user, can't change ownership of files he doesn't own. Since this un-tarballed script is owned by games, the chown command fails.
I used your Dockerfiles and got the same issue on MacOS.
You can get around this by, well, not using ADD. Which is funny because local tarball extraction is the one use case where Docker thinks you should prefer ADD over COPY.
COPY code/validation_utility.tar.gz ${HOMEDIR}/.
RUN tar -xvf validation_utility.tar.gz
Properly extracts the tarball and, since dsadmin was the user at the time, the contents come out properly owned by dsadmin.
(An uglier route might be to switch the USER to root to set permissions, then set it back to dsadmin. I think this is icky, but it's an option.)