I'm building Python 3.7.4 (It's a hard requirement for other software) on a base Ubuntu 20.04 image using a Dockerfile. I'm following this guide.
Everything works fine if I run the image and follow the guide, but I want to setup my virtual environment in the Dockerfile and have the pip requirements persistent when running the image.
Here's the relevant part of my Dockerfile:
...
RUN echo =============== Building and Install Python =============== \
&& cd /tmp \
&& wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz \
&& tar xvf ./Python-3.7.4.tgz \
&& cd Python-3.7.4 \
&& ./configure --enable-optimizations --with-ensurepip=install \
&& make -j 8 \
&& sudo make install
ENV VIRTUAL_ENV=/opt/python-3.7.4
ENV PATH="$VIRTUAL_ENV:$PATH"
COPY "./hourequirements.txt" /usr/local/
RUN echo =============== Setting up Python Virtual Environment =============== \
&& python3 -m venv $VIRTUAL_ENV \
&& source $VIRTUAL_ENV/bin/activate \
&& pip install --upgrade pip \
&& pip install --no-input -r /usr/local/hourequirements.txt
...
The Dockerfile builds without errors, but when I run the image the environment doesn't exist and python 3.7.4 doesn't show any of the installed requirements.
How can I install Python modules in the virtual environment using PIP in the Dockerfile and have them persist when the docker image runs?
Usual find answer just after post.
I changed:
ENV PATH="$VIRTUAL_ENV:$PATH"
to:
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
in Dockerfile and started working correctly.
I am trying to create a dockerfile for my NGS tools.
I am working on a cluster and I had installed miniconda3 locally.
Firstly I created a base dockerfile containing the path of miniconda bin and build it.
After that I created the dockerfile for my tool.
This is what my base docker file looks like:
FROM centos:7
ENV PATH="/home/nirmal/new-cluster/miniconda/bin:$PATH"
RUN conda install -y mamba -c conda-forge
RUN conda init bash
RUN echo "conda activate base" >> ~/.bashrc
And this is what my docker image for the tool looks like:
FROM base:latest
RUN echo "source activate mitozEnv" > ~/.bashrc
ENV PATH="/home/nirmal/new-cluster/miniconda/envs/mitozEnv/bin:$PATH"
VOLUME /nfs_master/nirmal/simulated_reads/25K/Hiseq
WORKDIR /nfs_master/nirmal/simulated_reads/25K/Hiseq
CMD ["/bin/bash"]
This is the script that I used:
fq1=sim_hiseq_exponential_R1.fastq
fq2=sim_hiseq_exponential_R1.fastq
outprefix=mitoz
time docker run -v $PWD:$PWD -w $PWD --rm mitoz:latest \
python3 /home/nirmal/tools/MitoZ/version_2.3/release_MitoZ_v2.3/MitoZ.py all \
--genetic_code 2 \
--clade Chordata \
--insert_size 300 \
--thread_number 4 \
--fastq1 $fq1 \
--fastq2 $fq2 \
--outprefix $outprefix \
--fastq_read_length 126 \
1>m.log 2>m.err
But when I tried running it with a small test data I got this error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "python3": executable file not found in $PATH: unknown.
Thanks in advance for your help.
I'm getting following error:
This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.
(Cloud assembly schema version mismatch: Maximum schema version supported is 8.0.0, but found 9.0.0)
after issuing cdk diff command.
I did run npm install -g aws-cdk#latest after which I've successfully installed new versions of packages: Successfully installed aws-cdk.assets-1.92.0 aws-cdk.aws-apigateway-1.92.0 aws-cdk.aws-apigatewayv2-1.92.0 ... etc. with pip install -r requirements.txt
However after typing cdk --version I'm still getting 1.85.0 (build 5f44668).
My part of setup.py is as follows:
install_requires=[
"aws-cdk.core==1.92.0",
"aws-cdk.aws-ec2==1.92.0",
"aws-cdk.aws_ecs==1.92.0",
"aws-cdk.aws_elasticloadbalancingv2==1.92.0"
],
And I'm stuck now, as downgrading packages setup.py to 1.85.0 throwing ImportError: cannot import name 'CapacityProviderStrategy' from 'aws_cdk.aws_ecs'.
Help :), I would like to use newest packages version.
I encountered this issue with a typescript package, after upgrading the cdk in package.json. As Maciej noted upgrading did not seem to work. I am installing the cdk cli with npm, and an uninstall followed by an install fixed the issue.
npm -g uninstall aws-cdk
npm -g install aws-cdk
So I've fixed it, but is too chaotic to describe the steps.
It seems like there are problems with the symlink
/usr/local/bin/cdk
which was pointing to version 1.85.0 and not the one I updated to 1.92.0.
I removed the aws-cdk from the node_modules and installed it again, then removed the symlink /usr/local/bin/cdk and recreated it manually with
ln -s /usr/lib/node_modules/aws-cdk/bin/cdk /usr/local/bin/cdk
nothing helps for mac OS except this command:
yarn global upgrade aws-cdk#latest
For those coming here that are not using a global install (using cdk from node_modules) and using a mono-repo. This issue is due to the aws-cdk package of the devDependencies not matching the version of the dependencies for the package.
I was using "aws-cdk": "2.18.0" in my root package.json but all my packages were using "aws-cdk-lib": "2.32.1" as their dependencies. By updating the root package.json to use "aws-cdk": "2.31.1" this solved the issue.
I have been experiencing this a few times as well, so i am just dropping this solution which helps me resolves version mismatches, particularly on existing projects.
For Python:
What you need to do is modify the setup.py to specify the latest version.
Either, implicitly
install_requires=[
"aws-cdk.core",
"aws-cdk.aws-ec2",
"aws-cdk.aws_ecs",
"aws-cdk.aws_elasticloadbalancingv2"
],
or explicitly;
install_requires=[
"aws-cdk.core==1.xx.x",
"aws-cdk.aws-ec2==1.xx.x",
"aws-cdk.aws_ecs==1.xx.x",
"aws-cdk.aws_elasticloadbalancingv2==1.xx.x"
],
Then from project root, run;
setup.py install
For TypeScript:
Modify package.json;
"dependencies": {
"#aws-cdk/core" : "latest",
"source-map-support": "^0.5.16"
}
Then run from project root:
npm install
I hope this helps! Please let me know if I need to elaborate or provide more details.
Uninstall the CDK version:
npm uninstall -g aws-cdk
Install specfic version which your application is using. For ex: CDK 1.158.0
npm install -g aws-cdk#1.158.0
If you've made it down here there are 2 options to overcome this issue.
Provision a cloud9 env and run the code there.
Run the app in a container.
Move the cdk app into a folder so that you have a parent folder to put a dockerfile a makefile and a dockercompose file.
fire up docker desktop
cd into the root folder from the terminal and then run make
Dockerfile contents
FROM ubuntu:20.04 as compiler
WORKDIR /app/
RUN apt-get update -y \
&& apt install python3 -y \
&& apt install python3-pip -y \
&& apt install python3-venv -y \
&& python3 -m venv venv
ARG NODE_VERSION=16
RUN ls
RUN apt-get update
RUN apt-get install xz-utils
RUN apt-get -y install curl
RUN apt-get update -y && \
apt-get upgrade -y && \
apt-get install -y && \
curl -sL https://deb.nodesource.com/setup_$NODE_VERSION.x | bash - && \
apt install -y nodejs
RUN apt-get install -y python-is-python3
RUN npm install -g aws-cdk
RUN python -m venv /opt/venv
docker-compose.yaml contents
version: '3.6'
services:
cdk-base:
build: .
image: cdk-base
command: ${COMPOSE_COMMAND:-bash}
volumes:
- .:/app
- /var/run/docker.sock:/var/run/docker.sock #Needed so a docker container can be run from inside a docker container
- ~/.aws/:/root/.aws:ro
Makefile content
SHELL=/bin/bash
CDK_DIR=cdk_app/
COMPOSE_RUN = docker-compose run --rm cdk-base
COMPOSE_UP = docker-compose up
PROFILE = --profile default
all: pre-reqs synth
pre-reqs: _prep-cache container-build npm-install
_prep-cache: #This resolves Error: EACCES: permission denied, open 'cdk.out/tree.json'
mkdir -p cdk_websocket/cdk.out/
container-build: pre-reqs
docker-compose build
container-info:
${COMPOSE_RUN} make _container-info
_container-info:
./containerInfo.sh
clear-cache:
${COMPOSE_RUN} rm -rf ${CDK_DIR}cdk.out && rm -rf ${CDK_DIR}node_modules
cli: _prep-cache
docker-compose run cdk-base /bin/bash
npm-install: _prep-cache
${COMPOSE_RUN} make _npm-install
_npm-install:
cd ${CDK_DIR} && ls && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt && npm -v && python --version
npm-update: _prep-cache
${COMPOSE_RUN} make _npm-update
_npm-update:
cd ${CDK_DIR} && npm update
synth: _prep-cache
${COMPOSE_RUN} make _synth
_synth:
cd ${CDK_DIR} && source .venv/bin/activate && pip install -r requirements.txt && cdk synth --no-staging ${PROFILE} && cdk deploy --require-approval never ${PROFILE}
bootstrap: _prep-cache
${COMPOSE_RUN} make _bootstrap
_bootstrap:
cd ${CDK_DIR} && source .venv/bin/activate && pip install -r requirements.txt && cdk bootstrap ${PROFILE}
deploy: _prep-cache
${COMPOSE_RUN} make _deploy
_deploy:
cd ${CDK_DIR} && cdk deploy --require-approval never ${PROFILE}
destroy:
${COMPOSE_RUN} make _destroy
_destroy:
cd ${CDK_DIR} && source .venv/bin/activate && pip install -r requirements.txt && cdk destroy --force ${PROFILE}
diff: _prep-cache
${COMPOSE_RUN} make _diff
_diff: _prep-cache
cd ${CDK_DIR} && cdk diff ${PROFILE}
test:
${COMPOSE_RUN} make _test
_test:
cd ${CDK_DIR} && npm test
This worked for me in my dev environment since I had re-cloned the source, I needed to re-run the npm install command. A sample package.json might look like this (update the versions as needed):
{
"dependencies": {
"aws-cdk": "2.27.0",
"node": "^16.14.0"
}
}
I changed typescript": "^4.7.3" version and it worked.
Note: cdk version 2("aws-cdk-lib": "^2.55.0")
I have a docker file like :
FROM conda/miniconda3-centos7
WORKDIR /tmp
COPY app/ /tmp
RUN conda install gcc_linux-64
RUN conda install gxx_linux-64
CMD ["python", "Hello_World.py"]
The code gets stuck after the first RUN conda command. The error i get is :
WARNING: The conda.compat module is deprecated and will be removed in a future release.
==> WARNING: A newer version of conda exists. <==
current version: 4.6.11
latest version: 4.9.2
Please update conda by running
$ conda update -n base -c defaults conda
Removing intermediate container 277edb28a107
---> e6b51d71eac0
Step 7/8 : RUN conda install gxx_linux-64
---> Running in 94166fbfff2a
Traceback (most recent call last):
File "/usr/local/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
The command '/bin/sh -c conda install gxx_linux-64' returned a non-zero code: 1
Can you please suggest?
Adding conda update -n base -c defaults conda in your Dockerfile solves the mentioned problem.
You could also consider using && for optimizing the creation of docker images. Read more about it here.
An optimized Dockerfile would be:
FROM conda/miniconda3-centos7
WORKDIR /arnav
COPY app/ /arnav
RUN conda update -n base -c defaults conda \
&& conda install gcc_linux-64 && conda install gxx_linux-64
CMD ["python", "Hello_World.py"]
I am trying to modify this docker compose setup to install a number of python packages including yfinance. In my mind there are three ways to install packages and i'd like to be able to use each one in docker:
install from pip requirements file
install from conda environments.yml file
install manually in environment by running install command (pip install yfinance or conda install -c conda-forge beautifulsoup4)
Here is a list of the problems I am having trying to modify this setup:
pip requirements file. - After altering this file the packages don't appear to be installed and instead the file is overridden with defaults on github.
conda environment file - Unable to create environment and install packages from environment.yml
manual install of package - entering a docker container shell using docker exec -it <containername> /bin/bash results in pip and conda returning command not found in bash.
Results so far:
All of the above methods have resulted in errors including 'command not found' or 'no module named yfinance' when importing in a notebook.
The only way I've been able to have any success so far is opening a notebook in the browser at localhost:8888 and creating a new notebook and running !pip install yfinance. However importing and executing the following code also results in an error making me think the package or dependencies haven't installed correctly.
import yfinance as yf
m = yf.Ticker("MSFT")
m.info
Here is my docker-compose file service
jupyter:
image: cmihai/jupyter:v1
container_name: 'joshnotebook'
hostname: jupyter
restart: 'always'
ports:
- '8888:8888'
env_file:
- notebook/config/jupyter.env
volumes:
- ${USERDIR}/docker/notebook:/home/user/notebook
build: ./notebook/services/jupyter
Here is my Dockerfile
FROM cmihai/python:v1
LABEL name="cmihai/jupyterlab" \
version="1" \
maintainer="Mihai Criveti <crivetimihai#gmail.com>" \
description="Anaconda Python with Jupyter Lab\
This container installs Anaconda Python 3 and Jupyter Lab."
SHELL ["/bin/bash", "-l", "-c"]
ENV DEBIAN_FRONTEND noninteractive
# PARAMETERS
ARG OS_PACKAGES="tzdata curl graphviz vim-nox gosu mc sqlite3 bash-completion sudo"
#ARG LATEX_PACKAGES="texlive texlive-latex-base texlive-latex-extra texlive-xetex texlive-generic-recommended texlive-fonts-extra texlive-fonts-recommended pandoc"
ARG CONDA_PACKAGES="jupyterlab pandas"
COPY condaenvironment.yml /tmp/condaenvironment.yml
# ROOT
USER root
# INSTALL OS PACKAGES
RUN apt-get update \
&& apt-get --no-install-recommends install --yes ${OS_PACKAGES} \ # ${LATEX_PACKAGES}\
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN conda install -c ranaroussi yfinance
# SWITCH TO REGULAR USER
RUN mkdir /home/user/notebook && chown -R user:user /home/user/notebook \
&& echo 'user ALL=NOPASSWD: ALL' >> /etc/sudoers
USER user
# INSTALL CONDA PACKAGES
RUN . ~/.profile \
&& unset SUDO_UID SUDO_GID SUDO_USER \
&& conda install --quiet --yes ${CONDA_PACKAGES} \
&& conda install -c anaconda requests \
&& conda install -c conda-forge lxml \
&& conda install -c conda-forge beautifulsoup4 \
&& conda install -c conda-forge sqlalchemy \
&& conda install -c anaconda mysql-connector-python \
&& conda install -c conda-forge seaborn \
&& conda install -c conda-forge altair \
&& conda clean -y -all
# INSTALL PIP PACKAGES
RUN . ~/.profile \
&& python -m pip install --no-cache-dir --upgrade pip \
&& python -m pip install yfinance
# WORKDIR
WORKDIR /home/user
# JUPYTERLAB EXTENSIONS
RUN jupyter labextension install jupyterlab-drawio \
&& jupyter labextension install jupyterlab_bokeh \
&& jupyter labextension install plotlywidget \
&& jupyter labextension install #jupyterlab/plotly-extension \
&& jupyter labextension install jupyterlab-chart-editor \
&& jupyter labextension install #jupyterlab/git \
&& jupyter serverextension enable --py jupyterlab_git \
&& jupyter labextension install #jupyterlab/latex \
&& jupyter labextension install #jupyterlab/toc \
&& jupyter labextension install #oriolmirosa/jupyterlab_materialdarker \
&& jupyter labextension install #jupyterlab/geojson-extension \
&& jupyter lab build
# COMMAND and ENTRYPOINT:
COPY start-jupyter.sh /home/user/start-jupyter.sh
CMD ["/home/user/start-jupyter.sh"]