I have this docker file based on alpine that installs several packages with conda. At the end installs pydrill with pip as there's no conda installation.
from jcrist/alpine-dask
RUN /opt/conda/bin/conda update -n base -c defaults conda -y
RUN /opt/conda/bin/conda update dask
RUN /opt/conda/bin/conda install -c conda-forge dask-ml
RUN /opt/conda/bin/conda install scikit-learn -y
RUN /opt/conda/bin/conda install flask -y
RUN /opt/conda/bin/conda install waitress -y
RUN /opt/conda/bin/conda install gunicorn -y
RUN /opt/conda/bin/conda install pytest -y
RUN /opt/conda/bin/conda install apscheduler -y
RUN /opt/conda/bin/conda install matplotlib -y
RUN /opt/conda/bin/conda install pyodbc -y
USER root
RUN apk update
RUN apk add py-pip
RUN pip install pydrill
When I build the docker image everything works fine. But when I run the container the command line starts gunicorn, but it fails with the following message:
File "/code/app/service/cm/exec/run_drill.py", line 1, in <module>
from pydrill.client import PyDrill
ModuleNotFoundError: No module named 'pydrill'
Is this pip installation correct? This is the docker compose:
version: "3.0"
services:
web:
image: img-dask
volumes:
- vol_py_code:/code
- vol_dask_data:/data
- vol_dask_model:/model
ports:
- "5000:5000"
working_dir: /code
environment:
- app.config=/code/conf/py.app.json
- common.config=/code/conf/py.common.json
entrypoint:
- /opt/conda/bin/gunicorn
command:
- -b 0.0.0.0:5000
- --reload
- app.frontend.app:app
scheduler:
image: img-dask
ports:
- "8787:8787"
- "8786:8786"
entrypoint:
- /opt/conda/bin/dask-scheduler
worker:
image: img-dask
depends_on:
- scheduler
environment:
- PYTHONPATH=/code
- MODEL_PATH=/model/rfc_model.pkl
- PREPROCESSING_PATH=/model/data_columns.pkl
- SCHEDULER_ADDRESS=scheduler
- SCHEDULER_PORT=8786
volumes:
- vol_py_code:/code
- vol_dask_data:/data
- vol_dask_model:/model
entrypoint:
- /opt/conda/bin/dask-worker
command:
- scheduler:8786
volumes:
vol_py_code:
name: vol_py_code
vol_dask_data:
name: vol_dask_data
vol_dask_model:
name: vol_dask_model
UPDATE
If I run the command line inside the container, I can see that pydrill is installed, but my code does not see the library.
/code/conf # pip3 list
Package Version
---------- ---------
certifi 2020.12.5
chardet 4.0.0
idna 2.10
pip 18.1
pydrill 0.3.4
requests 2.25.1
setuptools 40.6.2
urllib3 1.26.4
You are using pip version 18.1, however version 21.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The problem is that pydrill and all other conda packages are in different environments. When the server starts, it doesn't see pydrill, only conda packages.
To fix the issue install pip itself in conda's environment:
from jcrist/alpine-dask
USER root
RUN /opt/conda/bin/conda create -p /pyenv -y
RUN /opt/conda/bin/conda install -p /pyenv dask scikit-learn flask waitress gunicorn \
pytest apscheduler matplotlib pyodbc -y
RUN /opt/conda/bin/conda install -p /pyenv -c conda-forge dask-ml -y
RUN /opt/conda/bin/conda install -p /pyenv pip -y
RUN /pyenv/bin/pip install pydrill
Can you try conda install pip instead of apk
Something like
from jcrist/alpine-dask
WORKDIR /opt/conda/bin
RUN conda update -n base -c defaults conda -y
RUN conda update dask
RUN install -c conda-forge dask-ml
RUN conda install stickit-learn flask waitress gunicorn \
pytest apscheduler matplotlib pydobc pip -y
RUN pip install pydrill
I have packaged pydrill for conda-forge, so you can simply conda install -c conda-forge pydrill.
Related
My file structure is as follows:
Dockerfile
.gitlab-ci.yml
Here is my Dockerfile:
FROM python:3
RUN apt-get update && apt-get install make
RUN apt-get install -y python3-pip
RUN pip3 install --upgrade pip
RUN pip3 install pygdbmi
RUN pip3 install pyyaml
RUN pip3 install Path
And here is my .gitlab-ci.yml file:
test-job:
stage: test
image: runners:test-harness
script:
- cd test-harness
# - pip3 install pygdbmi
# - pip3 install pyyaml
- python3 main.py
artifacts:
untracked: false
when: on_success
expire_in: "30 days"
paths:
- test-harness/script.log
For some reason the pip3 install in the Dockerfile doesn't seem to be working as I get the error:
python3 main.py
Traceback (most recent call last):
File "/builds/username/test-harness/main.py", line 6, in <module>
from pygdbmi.gdbcontroller import GdbController
ModuleNotFoundError: No module named 'pygdbmi'
When I uncomment the two commented lines in .gitlab-ci.yml:
# - pip3 install pygdbmi
# - pip3 install pyyaml
It works fine but ideally, I want those 2 packages to be installed in the Dockerfile not the .gitlab-ci.yml pipeline stage
I've tried changing the WORKDIR as well as USER and it doesn't seem to have any effect.
Any ideas/solutions?
unfortunately I'm a bit desperate.
I have created a dockerimage and am running everything with docker-compose.
If i run docker-compose up i get this error:
| django.core.exceptions.ImproperlyConfigured: 'mssql' isn't an available database backend or couldn't be imported. Check the above exception. To use one of the built-in backends, use 'django.db.backends.XXX', where XXX is one of:
web_1 | 'mysql', 'oracle', 'postgresql', 'sqlite3'
if i see in the pip list, i see to less packages.
:(
docker-compose run web pip list
Package Version
---------- -------
asgiref 3.5.2
Django 4.0.4
pip 22.0.4
psycopg2 2.9.3
setuptools 58.1.0
sqlparse 0.4.2
wheel 0.37.1
´´´
Dockerfile
FROM ubuntu:20.04
FROM python:3
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN apt update -y && apt upgrade -< && apt-get update
RUN apt-get install -y pip curl git python3-pip openjdk-8-jdk unixodbc-dev
#RUN pip install --upgrade pip
RUN pip install -r requirements.txt
#ADD SQL SERVER ODBC Driver 17 for Ubuntu 20.04
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN ACCEPT_EULA=Y apt-get install -y --allow-unauthenticated msodbcsql17
RUN ACCEPT_EULA=Y apt-get install -y --allow-unauthenticated mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
COPY . /code/
´´´
requirements.txt
Django>=4.0
psycopg2>=2.8
django-mssql==1.8
djangorestframework==3.13.1
pymssql==2.2.3
pyodbc==4.0.32
pyparsing==3.0.4
setuptools==61.2.0
sqlparse==0.4.1
´´´
docker-compose.yml
version: "3"
services:
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
docker-compose up build -d
This builds the compose new. Then i get easy errors to fix. Just change some lines in Dockerfile.
Also change in requirements django-mssql into mssql-django
Why can't I map this docker command, which pip installs packages from a local network repository hosted at nexus.corp.com:
$> docker build -t demo --no-cache --add-host nexus.corp.com:1.2.3.4 .
which succeeds, into this docker-compose configuration:
version: "3"
services:
app:
build:
context: .
extra_hosts: ['nexus.corp.com:1.2.3.4']
command: >
sh -c "ping -c 4 nexus.corp.com"
which fails during the build step involving pip installing packages from the local repository?
Dockerfile
FROM python:3.8-slim
ENV PYTHONUNBUFFERED 1
# Install postgres client
RUN apt-get update
RUN apt-get install -y python3.8-dev
# For testing/debugging
RUN apt-get install -y iputils-ping
RUN pip install -U pip setuptools
WORKDIR /work
COPY ./pip.conf /etc/pip.conf # use custom pip config, see below
# Install a package hosted at the custom location
RUN pip3 install custom_package
pip.conf
[global]
index = https://nexus.corp.com/repository/corp-pypi-group/pypi
index-url = https://nexus.corp.com/repository/corp-pypi-group/simple
All of this networking is going on through a VPN, so the nexus.corp.com name isn't being served out by a DNS.
I am trying to install rabbitmq (pika) driver in my python container, but in local deployment, there is no problem.
FROM ubuntu:20.04
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN apt-get update && apt-get -y install gcc python3.7 python3-pip
RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt
COPY . .
CMD ["python","index.py"]
this is my requerments.txt file :
requests
telethon
Flask
flask-mongoengine
Flask_JWT_Extended
Flask_Bcrypt
flask-restful
flask-cors
jsonschema
werkzeug
pandas
xlrd
Kanpai
pika
Flask-APScheduler
docker build steps complete with no error and install all the dependencies with no error but when I try to run my container it crashes with this error :
no module named 'pika'
installing python3.7 will not work here, you are still using python3.8 by using pip3 command and your CMD will also start python3.8, I suggest you to use python:3.7 base image
so try this:
FROM python:3.7
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN apt-get update && apt-get -y install gcc
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
COPY . .
CMD ["python","index.py"]
I'm getting the following error when executing
docker-compose up --build
web_1 | Traceback (most recent call last):
web_1 | File "glm-plotter.py", line 4, in <module>
web_1 | from flask import Flask, render_template, request, session
web_1 | ModuleNotFoundError: No module named 'flask'
glm-plotter_web_1 exited with code 1
I tried changing "Flask" to "flask" in the requirements.txt
Dockerfile
FROM continuumio/miniconda3
RUN apt-get update && apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y build-essential
COPY requirements.txt /
RUN pip3 install --trusted-host pypi.python.org -r /requirements.txt
ADD ./glm-plotter /code
WORKDIR /code
RUN ls .
CMD ["python3", "glm-plotter.py"]
docker-compose.yml
version: "3"
services:
web:
volumes:
- ~/.aws:/root/.aws
build: .
ports:
- "5000:5000"
requirements.txt
click==6.6
Flask==0.11.1
itsdangerous==0.24
Jinja2==2.8
MarkupSafe==0.23
numpy==1.11.1
pandas==0.18.1
python-dateutil==2.5.3
pytz==2016.4
six==1.10.0
Werkzeug==0.11.10
glm-plotter.py
from flask import Flask, render_template, request, session
import os, json
import GLMparser
...
I created a Docker and it compiles fine. You might want to adapt it to your personal needs and add the last few lines from your Dockerfile above:
FROM library/python:3.6-stretch
COPY requirements.txt /
RUN pip install -r /requirements.txt
Please note that in the library/python images no explicit version number for python or pip is required since there is only one installed.
If you use miniconda image you have to create a new environment and activate it prior to installing the packages and run the program in your docker file. Something like:
FROM continuumio/miniconda3
RUN apt-get update && apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y build-essential
COPY requirements.txt /
RUN ["conda", "create", "-n", "myenv", "python=3.4"]
RUN /bin/bash -c "source activate myenv && pip install --trusted-host pypi.python.org -r /requirements.txt"
ADD ./glm-plotter /code
WORKDIR /code
RUN ls .
CMD /bin/bash -c "source activate myenv && python glm-plotter.py"