I have been trying to customize the default CKAN docker image to include my prefered extensions. The problem is that the precedence of installing pip packages, configuring the ini files and, building and running the main CKAN image is not clear for me.
I have tried adding a layer to the Dockerfile (see below) but extensions are not available after running docker-compose up. The add-extensions.sh file contains pip install commands for each extension.
RUN sh $CKAN_VENV/src/ckan/contrib/docker/add-extensions.sh
I have also tried to include my commands in the docker-compose file itself, inside the ckan service, as follows.
command: >
sh -c "
pip install ckanext-geoview &&
pip install ckanext-datarequests &&
ckan config-tool "/etc/ckan/production.ini" -f "/etc/ckan/custom-config.ini" &&
ckan config-tool "/etc/ckan/production.ini" -s app:main -e ckan.plugins='stats text_view image_view recline_view datastore datapusher resource_proxy geo_view datarequests' &&
ckan -c /etc/ckan/production.ini run --host 0.0.0.0"
But this is not working either. So what is the recommended way of including custom plugins inside the default CKAN image?
Related
Below my docker file,
FROM python:3.9.0
ARG WORK_DIR=/opt/quarter_1
RUN apt-get update && apt-get install cron -y && apt-get install -y default-jre
# Install python libraries
COPY requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip && pip install -r /tmp/requirements.txt
WORKDIR $WORK_DIR
EXPOSE 8888
VOLUME /home/data/quarter_1/
# Copy etl code
# copy code on container under your workdir "/opt/quarter_1"
COPY . .
I tried to connect to the server then i did the build with docker build -t my-python-app .
when i tried to run the container from a build image i got nothing and was not able to do it.
docker run -p 8888:8888 -v /home/data/quarter_1/:/opt/quarter_1 image_id
work here is opt
Update based on comments
If I understand everything you've posted correctly, my suggestion here is to use a base Docker Jupyter image, modify it to add your pip requirements, and then add your files to the work path. I've tested the following:
Start with a dockerfile like below
FROM jupyter/base-notebook:python-3.9.6
COPY requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip && pip install -r /tmp/requirements.txt
COPY ./quarter_1 /home/jovyan/quarter_1
Above assumes you are running the build from the folder containing dockerfile, "requirements.txt", and the "quarter_1" folder with your build files.
Note "home/joyvan" is the default working folder in this image.
Build the image
docker build -t biwia-jupyter:3.9.6 .
Start the container with open port to 8888. e.g.
docker run -p 8888:8888 biwia-jupyter:3.9.6
Connect to the container to access token. A few ways to do but for example:
docker exec -it CONTAINER_NAME bash
jupyter notebook list
Copy the token in the URL and connect using your server IP and port. You should be able to paste the token there, and afterwards access the folder you copied into the build, as I did below.
Jupyter screenshot
If you are deploying the image to different hosts this is probably the best way to do it using COPY/ADD etc., but otherwise look at using Docker Volumes which give you access to a folder (for example quarter_1) from the host, so you don't constantly have to rebuild during development.
Second edit for Python 3.9.0 request
Using the method above, 3.9.0 is not immediately available from DockerHub. I doubt you'll have much compatibility issues between 3.9.0 and 3.9.6, but we'll build it anyway. We can download the dockerfile folder from github, update a build argument, create our own variant with 3.9.0, and do as above.
Assuming you have git. Otherwise download the repo manually.
Download the Jupyter Docker stack repo
git clone https://github.com/jupyter/docker-stacks
change into the base-notebook directory of the cloned repo
cd ./base-notebook
Build the image with python 3.9.0 instead
docker build --build-arg PYTHON_VERSION=3.9.0 -t jupyter-base-notebook:3.9.0 .
Create the version with your copied folders and 3.9.0 version from the steps above, replacing the first line in the dockerfile instead with:
FROM jupyter-base-notebook:3.9.0
I've tested this and it works, running Python 3.9.0 without issue.
There are lots of ways to build Jupyter images, this is just one method. Check out docker hub for Jupyter to see their variants.
I've installed and configured docker (as per documentation) and I am trying to build a flask application using tiangolo/uwsgi-nginx-flask:python3.8. I've built a hello-world application, and have tested it locally by running python manage.py and the application runs successfully. Link to full Code-File.
My docker version and installation is as below:
Dockerfile:
FROM tiangolo/uwsgi-nginx-flask:python3.8
ENV INSTALL_PATH /usr/src/helloworld
RUN mkdir -p $INSTALL_PATH
# install net-tools
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y \
net-tools \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# set working directory
WORKDIR $INSTALL_PATH
# setup flask environment
# install all requirements
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
# copy all files and folder to docker
COPY . .
# run the application in docker environment
CMD [ "python", "./manage.py" ]
I built the application with docker build --tag hello-world:test . and running the application as: docker run -d -p 5000:5000 hello-world:test successfully.
However, I'm unable to open the application in localhost:5000 or 0.0.0.0:5000 or any other port. The application is running, as I can see it from the CLI:
But, from browser the page is not reachable:
The question mentions to check the IP address:
docker inspect -f "{{ .NetworkSettings.Networks.nat.IPAddress }}" hungry_engelbart
>> <no value>
Found another solution at this link, but docker-machine is currently deprecated.
I'm new to docker, but I have tried to run the same thing following this tutorial, but faced similar issues.
Finally, I am able to solve this. I had to configure a new inbound rules under Windows Firewall > Advanced Settings > Inbound Rules > New Inbound Rules. Create a new rule that will allow a range of local IP addresses, which in my case was 198.168.0.1:198.168.0.100. Finally, you need to run the application at 0.0.0.0 as pointed by #tentative in the comments. :)
I'm trying to implement Deploy to Heroku functionality for my Python application:
https://github.com/jet-admin/jet-bridge/tree/heroku
It woks OK if just use requirements.txt to install dependencies, but it require me to modify my requirements.txt to include some extra packages that i normally don't need (psycopg2, mysqlclient).
Is it possible not to include all requirements in requirements.txt, but install it with some extra command? I've tried adding postdeploy script which will perform pip install command, but after deploy succed my application says that psycopg2 is not installed (thought i installed it in postdeploy command).
The Heroku Python buildpack has a hook where you can execute extra commands after the initial slug compilation.
To use it, you can add a bin/post_compile file, putting inside the shell commands for installing the extra packages.
You can even make it depend on an environment variable, like:
# assuming you have files mysql-requirements.txt and postgres-requirements.txt
if [ "$JET_BRIDGE_DB" == "mysql" ]; then
echo "Installing Python dependencies for MySQL support"
pip install -r mysql-requirements.txt
else
echo "Assuming Postgres database, installing Python dependencies"
pip install -r postgres-requirements.txt
fi
Read more:
Buildpacks: https://devcenter.heroku.com/articles/buildpacks
Heroku Python Buildpack: https://github.com/heroku/heroku-buildpack-python
Slug compilation: https://devcenter.heroku.com/articles/slug-compiler
I have created a Python command line application that is available through PyPi / pip install.
The application has native dependencies.
To make the installation less painful for Windows users I would like to create a Dockerised version out of this command line application.
What are the steps to convert setup.py with an entry point and requirements.txt to a command line application easily? Are there any tooling around this, or should I just write Dockerfile by hand?
Well, You have to create a Dockerfile and build an image off of it. There are best practices regarding the docker image creation that you need to apply. There are also language specific best practices.
Just to give you some ideas about the process:
FROM python:3.7.1-alpine3.8 #base image
ADD . /myapp # add project files
WORKDIR /myapp
RUN apk add dep1 dep2 #put your dependency packages here
RUN pip-3.7 install -r requirements.txt #install pip packages
RUN pip-3.7 install .
CMD myapp -h
Now build image and push it to some public registry:
sudo docker build -t <yourusername>/myapp:0.1 .
users can just pull image and use it:
sudo docker run -it myapp:0.1 myapp.py <switches/arguments>
Some times i need to use modules which are not a part of default python installation and some times even packages like Anaconda or Canopy does not include them. So every time I move my project to another machine or just reinstall python i need to download them again. So my question is. Is there a way to store necessory modules in the project folder and use them from it without moving to default python installation folder.
You can use virtual environment or docker to install the required modules in your project dir so it is isolated from your system Python installation. In fact, you don't need Python installed on your machine when using docker.
Here is my workflow when developing Django web app with Docker. If your project dir is in /Projects/sampleapp, change the current working directory to the project dir and run the following.
Run a docker container from your terminal:
docker run \
-it --rm \
--name django_app \
-v ${PWD}:/app \
-w /app \
-e PYTHONUSERBASE=/app/.vendors \
-p 8000:8000 \
python:3.5 \
bash -c "export PATH=\$PATH:/app/.vendors/bin && bash"
# Command expalanation:
#
# docker run Run a docker container
# -it Set interactive and allocate a pseudo-TTY
# -rm Remove the container on exit
# --name django_app Set the container name
# -v ${PWD}:/app Mount current dir as /app in the container
# -w /app Set the current working directory to /app
# -e PYTHONUSERBASE=/app/.vendors pip will install packages to /app/.vendors
# -p 8000:8000 Open port 8000
# python:3.5 Use the Python:3.5 docker image
# bash -c "..." Add /app/.vendors/bin to PATH and open the shell
On the container's shell, install the required packages:
pip install django celery django-allauth --user
pip freeze > requirements.txt
The --user options along with the PYTHONUSERBASE environment variable will make pip installs the packages in /app/.vendors.
Create the django project and develop the app as usual:
django-admin startproject sampleapp
cd sampleapp
python manage.py runserver 0.0.0.0:8000
The directory structure will look like this:
Projects/
sampleapp/
requirements.txt
.vendors/ # Note: don't add this dir to your VCS
sampleapp/
manage.py
...
This configuration enables you to install the packages in your project dir, isolated from your system. Note that you need to add requirements.txt to your VCS, but remember to exclude .vendors/ dir.
When you need to move and run the project on another machine, run the docker command above and reinstall the required packages on the container's shell:
pip install -r requirements.txt --user