How can I develop outside a docker container but still have nosetests watch my file changes and rerun my unit tests inside the container?
Here's my Dockerfile
FROM ubuntu
# Install Python.
RUN \
apt-get update && \
apt-get install -y python python-dev python-pip python-virtualenv && \
rm -rf /var/lib/apt/lists/* && \
pip install nose nose-watch mock && \
locale-gen en_US.UTF-8
# Define working directory.
WORKDIR /data/test/src
# Define default command.
CMD ["bash"]
Here are my commands:
docker build -t="test” .
docker run -it -v ~/test/src:/data/test/src test
When I run nosetests --with-watch inside the container, everything works. However, if I make file changes outside the container (I want to develop outside the container), then nosetests won't detect the changes and rerun the tests. I thought a volume was supposed to share files from host to container...
Related
I'm currently pretty stuck finding a solution for the following error:
LibreOfficeError: [Java framework] Error in function createSettingsDocument (elements.cxx).
javaldx failed!
Warning: failed to read path from javaldx
I start libreOffice in headless mode with subprocess.run from a Python / gunicorn application, for converting docx into pdf files:
args = ['/usr/lib64/libreoffice/program/soffice', '--headless', '--convert-to', 'pdf', '--outdir', pdfDocFolder, tmpDocName]
process = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
The error message above is what I get when trying to start the conversion.
My application runs in a docker container. The odd thing is that it worked out pretty well previously, when I used the S2I build process of OpenShift to build and deploy the image. Now, after abandoning S2I, building the image locally, and deploying it on OpenShift, I get that error message. I found some discussions of the very same error message in other contexts, stating that the working directory must be made writeable for non-root users and exported as HOME. Unfortunately, that didn't make a difference. I made the working dir writeable for all users. HOME is set to the correct directory. There must be some difference in the S2I build process compared to a local docker build, which makes a difference permission-wise.
That's the two Dockerfiles I use for building the image locally:
Base image:
FROM centos/python-36-centos7
EXPOSE 8080
USER root
RUN yum -y --disablerepo=\* --enablerepo=base,updates update && \
yum -y install libreoffice && \
yum -y install unoconv && \
yum -y install cairo && \
yum -y install cups-libs && \
yum -y install java-1.8.0-openjdk && \
yum clean all -y && \
rm -rf /var/cache/yum
RUN chown 1001:0 /usr/bin/soffice && \
chown 1001:0 /usr/share/fonts/local && \
chown -R 1001:0 /usr/lib64/libreoffice && \
fix-permissions /usr/lib64/libreoffice -P && \
rpm-file-permissions
USER 1001
And that's the Dockerfile built on top of the base image:
ARG REGISTRY_PATH=
ARG BRANCH_NAME=
FROM $REGISTRY_PATH:$BRANCH_NAME-latest
USER root
ENV APP_ROOT=/opt/app-root
ENV PATH=${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT}/src
COPY src ${APP_ROOT}/src
RUN pip install -r requirements.txt
RUN mkdir -p ${APP_ROOT}/.config/libreoffice/4/user && \
chmod -R a+rwx ${APP_ROOT}/src && \
chgrp -R 0 ${APP_ROOT}/src && \
chmod -R g=u ${APP_ROOT}/src /etc/passwd
EXPOSE 8080
USER 1001
WORKDIR ${APP_ROOT}/src
CMD ["gunicorn", "wsgi", "--bind", "0.0.0.0:8080", "--config", "config.py"]
Some hints or ideas to try out would really help me, since I completely ran out of options to pursue.
Thanks a lot.
I know this comes too late, but I was experiencing the same issue on a SpringBoot microservice I wrote recently, and for me the solution was to set the HOME env variable to a folder assigned to the group "0", since Openshift runs containers with random users for security reasons, but all of them belonging to the '0' group. Here's my dockerfile:
FROM docker.io/eclipse-temurin:17.0.5_8-jre-alpine
RUN apk update && apk upgrade --no-cache
RUN apk add libreoffice
RUN chgrp -R 0 /tmp && chmod -R g=u /tmp
ENV HOME /tmp
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["sh", "-c", "java ${JAVA_OPTS} -jar /app.jar"]
The most important instruction is the
RUN chgrp -R 0 /tmp && chmod -R g=u /tmp
assigning the right group to the folder which I will set as the HOME path. This kind of directive helped me solve several problems when migrating docker containers to Openshift and it's suggested by redhat itself in the official guide to build images:
https://docs.openshift.com/container-platform/4.9/openshift_images/create-images.html
hope this helps! :)
I have built a docker image using a Dockerfile that does the following:
FROM my-base-python-image
WORKDIR /opt/data/projects/project/
RUN mkdir files
RUN rm -rf /etc/yum.repos.d/*.repo
COPY rss-centos-7-config.repo /etc/yum.repos.d/
COPY files/ files/
RUN python -m venv /opt/venv && . /opt/venv/activate
RUN yum install -y unzip
WORKDIR files/
RUN unzip file.zip && rm -rf file.zip && . /opt/venv/bin/activate && python -m pip install *
WORKDIR /opt/data/projects/project/
That builds an image that allows me to run a custom command. In a terminal, for instance, here is the commmand I run after activating my project venv:
python -m pathA.ModuleB -a inputfile_a.json -b inputfile_b.json -c
Arguments a & b are custom tags to identify input files. -c calls a block of code.
So to run the built image successfully, I run the container and map local files to input files:
docker run --rm -it -v /local/inputfile_a.json:/opt/data/projects/project/inputfile_a.json -v /local/inputfile_b.json:/opt/data/projects/project/inputfile_b.json image-name:latest bash -c 'source /opt/venv/bin/activate && python -m pathA.ModuleB -a inputfile_a.json -b inputfile_b.json -c'
Besides shortening file paths, is there anythin I can do to shorten the docker run command? I'm thinking that adding a CMD and/or ENTRYPOINT to the Dockerfile would help, but I cannot figure out how to do it as I get errors.
There are a couple of things you can do to improve this.
The simplest is to run the application outside of Docker. You mention that you have a working Python virtual environment. A design goal of Docker is that programs in containers can't generally access files on the host, so if your application is all about reading and writing host files, Docker may not be a good fit.
Your file paths inside the container are fairly long, and this is bloating your -v mount options. You don't need an /opt/data/projects/project prefix; it's very typical just to use short paths like /app or /data.
You're also installing your application into a Python virtual environment, but inside a Docker image, which provides its own isolation. As you're seeing in your docker run command and elsewhere, the mechanics of activating a virtual environment in Docker are a little hairy. It's also not necessary; just skip the virtual environment setup altogether. (You can also directly run /opt/venv/bin/python and it knows it "belongs to" a virtual environment, without explicitly activating it.)
Finally, in your setup.py file, you can use a setuptools entry_points declaration to provide a script that runs your named module.
That can reduce your Dockerfile to more or less
FROM my-base-python-image
# OS-level setup
RUN rm -rf /etc/yum.repos.d/*.repo
COPY rss-centos-7-config.repo /etc/yum.repos.d/
RUN yum install -y unzip
# Copy the application in
WORKDIR /app/files
COPY files/ ./
RUN unzip file.zip \
&& rm file.zip \
&& pip install *
# Typical runtime metadata
WORKDIR /app
CMD main-script --help
And then when you run it, you can:
docker run --rm -it \
-v /local:/data \ # just map the entire directory
image-name:latest \
main-script -a /data/inputfile_a.json -b /data/inputfile_b.json -c
You can also consider the docker run -w /data option to change the current directory, which would add a Docker-level argument but slightly shorten the script command line.
I have installed a library called fastai==1.0.59 via requirements.txt file inside my Dockerfile.
But the purpose of running the Django app is not achieved because of one error. To solve that error, I need to manually edit the files /site-packages/fastai/torch_core.py and site-packages/fastai/basic_train.py inside this library folder which I don't intend to.
Therefore I'm trying to copy the fastai folder itself from my host machine to the location inside docker image.
source location: /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/
destination location: ../venv/lib/python3.6/site-packages/ which is inside my docker image.
being new to docker, I tried this using COPY command like:
COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/
which gave me an error:
ERROR: Service 'app' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder583041406/Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai: no such file or directory.
I tried referring this: How to include files outside of Docker's build context?
but seems like it bounced off my head a bit..
Please help me tackling this. Thanks.
Dockerfile:
FROM python:3.6-slim-buster AS build
MAINTAINER model1
ENV PYTHONUNBUFFERED 1
RUN python3 -m venv /venv
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y git && \
apt-get install -y build-essential && \
apt-get install -y awscli && \
apt-get install -y unzip && \
apt-get install -y nano && \
apt-get install -y libsm6 libxext6 libxrender-dev
RUN apt-cache search mysql-server
RUN apt-cache search libmysqlclient-dev
RUN apt-get install -y libpq-dev
RUN apt-get install -y postgresql
RUN apt-cache search postgresql-server-dev-9.5
RUN apt-get install -y libglib2.0-0
RUN mkdir -p /model/
COPY . /model/
WORKDIR /model/
RUN pip install --upgrade awscli==1.14.5 s3cmd==2.0.1 python-magic
RUN pip install -r ./requirements.txt
EXPOSE 8001
RUN chmod -R 777 /model/
COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/
CMD python3 -m /venv/activate
CMD /model/my_setup.sh development
CMD export API_ENV = development
CMD cd server && \
python manage.py migrate && \
python manage.py runserver 0.0.0.0:8001
Short Answer
No
Long Answer
When you run docker build the current directory and all of its contents (subdirectories and all) are copied into a staging area called the 'build context'. When you issue a COPY instruction in the Dockerfile, docker will copy from the staging area into a layer in the image's filesystem.
As you can see, this procludes copying files from directories outside the build context.
Workaround
Either download the files you want from their golden-source directly into the image during the build process (this is why you often see a lot of curl statements in Dockerfiles), or you can copy the files (dirs) you need into the build-tree and check them into source control as part of your project. Which method you choose is entirely dependent on the nature of your project and the files you need.
Notes
There are other workarounds documented for this, all of them without exception break the intent of 'portability' of your build. The only quality solutions are those documented here (though I'm happy to add to this list if I've missed any that preserve portability).
I have a script python which should output a file csv. I'm trying to have this file in the current working directory but without success.
This is my Dockerfile
FROM python:3.6.4
RUN apt-get update && apt-get install -y libaio1 wget unzip
WORKDIR /opt/oracle
RUN wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-
basiclite-linuxx64.zip && \ unzip instantclient-basiclite-linuxx64.zip && rm
-f instantclient-basiclite-linuxx64.zip && \ cd /opt/oracle/instantclient*
&& rm -f jdbc occi mysql *README jar uidrvci genezi adrci && \ echo
/opt/oracle/instantclient > /etc/ld.so.conf.d/oracle-instantclient.conf &&
ldconfig
RUN pip install --upgrade pip
COPY . /app
WORKDIR /app
RUN pip install --upgrade pip
RUN pip install pystan
RUN apt-get -y update && python3 -m pip install cx_Oracle --upgrade
RUN pip install -r requirements.txt
CMD [ "python", "Main.py" ]
And run the container with the following command
docker container run -v $pwd:/home/learn/rstudio_script/output image
This is bad practice to bind a volume just to have 1 file on your container be saved onto your host.
Instead, what you should leverage is the copy command:
docker cp <containerId>:/file/path/within/container /host/path/target
You can set this command to auto execute with bash, after your docker run.
So something like:
#!/bin/bash
# this stores the container id
CONTAINER_ID=$(docker run -dit img)
docker cp $CONTAINER_ID:/some_path host_path
If you are adamant on using a bind volume, then as the others have pointed out, the issue is most likely your python script isn't outputting the csv to the correct path.
Your script Main.py is probably not trying to write to /home/learn/rstudio_script/output. The working directory in the container is /app because of the last WORKDIR directive in the Dockerfile. You can override that at runtime with --workdir but then the CMD would have to be changed as well.
One solution is to have your script write files to /output/ and then run it like this:
docker container run -v $PWD:/output/ image
FROM ubuntu:14.04.2
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -y update && apt-get upgrade -y
RUN apt-get install python build-essential python-dev python-pip python-setuptools -y
RUN apt-get install libxml2-dev libxslt1-dev python-dev -y
RUN apt-get install libpq-dev postgresql-common postgresql-client -y
RUN apt-get install openssl openssl-blacklist openssl-blacklist-extra -y
RUN apt-get install nginx -y
RUN pip install "pip>=7.0"
RUN pip install virtualenv uwsgi
ADD canonicaliser_api /home/ubuntu/canonicaliser_api
ADD config_local.py /home/ubuntu/canonicaliser_api/config/config_local.py
RUN virtualenv /home/ubuntu/canonicaliser_api/venv
RUN source /home/ubuntu/canonicaliser_api/venv/bin/activate && pip install -r /home/ubuntu/canonicaliser_api/requirements.txt
RUN export CFLAGS=-I/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include/
RUN source /home/ubuntu/canonicaliser_api/venv/bin/activate && cd /home/ubuntu/canonicaliser_api/canonicaliser/cython_extensions/ && python setup.py build_ext --inplace
RUN cp /home/ubuntu/canonicaliser_api/canonicaliser/cython_extensions/canonicaliser/cython_extensions/*.so /home/ubuntu/canonicaliser_api/canonicaliser/cython_extensions
RUN rm -rf /home/ubuntu/canonicaliser_api/canonicaliser/cython_extensions/canonicaliser
RUN rm -r /home/ubuntu/canonicaliser_api/canonicaliser/cython_extensions/build
RUN mkdir /var/run/flask-uwsgi
RUN chown -R www-data:www-data /var/run/flask-uwsgi
RUN mkdir /var/log/flask-uwsgi
RUN touch /var/log/flask-uwsgi/dqs_canon.log
RUN chown -R www-data:www-data /var/log/flask-uwsgi
RUN mkdir /etc/flask-uwsgi
ADD configs/new-canon/flask-uwsgi/flask-uwsgi.conf /etc/init/
ADD configs/new-canon/flask-uwsgi/flask-uwsgi.ini /etc/flask-uwsgi/
EXPOSE 8888
CMD service flask-uwsgi restart
# RUN echo "daemon off;" >> /etc/nginx/nginx.conf
# CMD service nginx start
When I try to run this docker I get the error message:
flask-uwsgi: unrecognized service
So I ended up uncommenting the last two lines, so that nginx gets started and keeps the docker process alive. I then ssh'ed into it to debug.
docker exec -it 20b2ff3a4cac bash
Now when I try to run the service, it is the same problem and I can't find any missing step. Maybe services are not allowed to be started like this in docker?
root#30b2ff3a4cac:/# service flask-uwsgi start
flask-uwsgi: unrecognized service
/etc/flask-uwsgi/flask-uwsgi.ini:
[uwsgi]
socket = /var/run/flask-uwsgi/flask-uwsgi.sock
home = /home/ubuntu/canonicaliser_api/venv
wsgi-file = flask_uwsgi.py
callable = app
master = true
; www-data uid/gid
uid = 33
gid = 33
http-socket = :8888
die-on-term = true
processes = 4
threads = 2
logger = file:/var/log/flask-uwsgi/flask-uwsgi.log
/etc/init/flask-uwsgi.conf:
start on [2345]
stop on [06]
script
cd /home/ubuntu/canonicaliser_api
exec uwsgi --ini /etc/flask-uwsgi/flask-uwsgi.ini
end script
While ssh'ed into the process, I could run the uwsgi like this directly and it works:
exec uwsgi --ini /etc/flask-uwsgi/flask-uwsgi.ini
So services are not supported in docker and I have to run this directly in docker image like this:
RUN exec uwsgi --ini /etc/flask-uwsgi/flask-uwsgi.ini
Or I'm missing something.
Yeah, don't use services.
You can't do this though:
RUN exec uwsgi --ini /etc/flask-uwsgi/flask-uwsgi.ini
That line will complete, and be committed to an image. But the process will no longer be running in subsequent instructions or when the container is started.
Instead, you can do it in an ENTRYPOINT or CMD command, as they are executed when the container starts. This should work:
CMD uwsgi --ini /etc/flask-uwsgi/flask-uwsgi.ini
Some other points:
you might find things easier if you use one of the official python images.
I would just get rid of virtualenv; I don't see the benefit of virtualenv in an isolated container.
Running RUN rm -rf ... doesn't save any space; those files were already committed to a previous layer. You need to delete files in the same instruction they are added to avoid using space in the image.
It might make sense to do USER www-data rather than chowning files.