I want to use debian:bullseye as a base image and then install a specific Python version - i.e. 3.11.1. At the moment I am just learning docker and linux.
From what I understand I can either:
Download and compile sources
Install binaries (using apt-get)
Use a Python base image
I have come across countless questions on here and articles online. Do I use deadsnakes? What version do I need? Are there any official python distributions (who is deadsnakes anyway)?
But ultimately I want to know the best means of getting Python on there. I don't want to use a Python base image - I am curious in the steps involved. Compile sources - I am far from having that level of knowhow - and one for another day.
Currently I am rolling with the following:
FROM debian:bullseye
RUN apt update && apt upgrade -y
RUN apt install software-properties-common -y
RUN add-apt-repository "ppa:deadsnakes/ppa"
RUN apt install python3.11
This fails with:
#8 1.546 E: Unable to locate package python3.11
#8 1.546 E: Couldn't find any package by glob 'python3.11'
Ultimately - it's not the error - its just finding a good way of getting a specific Python version on my container.
In case you want to install Python 3.11 in debian bullseye you have to compile it from source following the next steps (inside the Dockerfile):
sudo apt update
sudo apt install software-properties-common wget
wget https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz
sudo tar -xf Python-3.11.1.tar.xz
cd Python-3.11.1
sudo ./configure --enable-optimizations
sudo make altinstall
Another option (easiest) would be to use the official Python Docker image, in your case:
FROM 3.11-bullseye
You have all the versions available in docker hub.
Other option that could be interesting in your case is 3.11-slim-bullseye, that is an image that does not contain the common packages contained in the default tag and only contains the minimal packages needed to run python.
Based on #tomasborella answer, to do this in docker:
Dockerfile
FROM debian:bullseye
RUN apt-get update -y \
&& apt-get upgrade -y \
&& apt-get -y install build-essential \
zlib1g-dev \
libncurses5-dev \
libgdbm-dev \
libnss3-dev \
libssl-dev \
libreadline-dev \
libffi-dev \
libsqlite3-dev \
libbz2-dev \
wget \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-get purge -y imagemagick imagemagick-6-common
RUN cd /usr/src \
&& wget https://www.python.org/ftp/python/3.11.0/Python-3.11.0.tgz \
&& tar -xzf Python-3.11.0.tgz \
&& cd Python-3.11.0 \
&& ./configure --enable-optimizations \
&& make altinstall
RUN update-alternatives --install /usr/bin/python python /usr/local/bin/python3.11 1
update-alternatives - will update the links to allow you to run python as opposed to specifying python3.11 when you want to run it.
It takes a while to compile those sources!
I've been using pyunpack.Archive().extractall(tempdir) successfully in a Cloud Run instance to extract .tar.gz and .zip files to a tempfile.TemporaryDirectory, but when I try the same approach with .rar files, I just get an empty temporary directory.
The strange thing is that the code works when run locally (On Ubuntu 20.04).
I have been wondering whether it has something to do with the system installation of the linux rar/ unrar binaries. I only managed to intall unrar-free using Docker. When trying to install unrar or rar, I get "No installation candidate" errors, despite adding the Multiverse ppa.
There is no error output when extracting the rar file, it just doesn't result in any output. I have checked the integrity of the rar file as well.
SOLVED it by adding the non-free repositories and installing both unrar and unrar-free
RUN apt-get update && \
apt-get install -y software-properties-common && \
apt-get update && \
#apt-get install -y python-software-properties && \
#apt-get update && \
apt-add-repository non-free
RUN apt-get update && \
apt-get install -y \
unrar-free \
unrar
I have installed a library called fastai==1.0.59 via requirements.txt file inside my Dockerfile.
But the purpose of running the Django app is not achieved because of one error. To solve that error, I need to manually edit the files /site-packages/fastai/torch_core.py and site-packages/fastai/basic_train.py inside this library folder which I don't intend to.
Therefore I'm trying to copy the fastai folder itself from my host machine to the location inside docker image.
source location: /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/
destination location: ../venv/lib/python3.6/site-packages/ which is inside my docker image.
being new to docker, I tried this using COPY command like:
COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/
which gave me an error:
ERROR: Service 'app' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder583041406/Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai: no such file or directory.
I tried referring this: How to include files outside of Docker's build context?
but seems like it bounced off my head a bit..
Please help me tackling this. Thanks.
Dockerfile:
FROM python:3.6-slim-buster AS build
MAINTAINER model1
ENV PYTHONUNBUFFERED 1
RUN python3 -m venv /venv
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y git && \
apt-get install -y build-essential && \
apt-get install -y awscli && \
apt-get install -y unzip && \
apt-get install -y nano && \
apt-get install -y libsm6 libxext6 libxrender-dev
RUN apt-cache search mysql-server
RUN apt-cache search libmysqlclient-dev
RUN apt-get install -y libpq-dev
RUN apt-get install -y postgresql
RUN apt-cache search postgresql-server-dev-9.5
RUN apt-get install -y libglib2.0-0
RUN mkdir -p /model/
COPY . /model/
WORKDIR /model/
RUN pip install --upgrade awscli==1.14.5 s3cmd==2.0.1 python-magic
RUN pip install -r ./requirements.txt
EXPOSE 8001
RUN chmod -R 777 /model/
COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/
CMD python3 -m /venv/activate
CMD /model/my_setup.sh development
CMD export API_ENV = development
CMD cd server && \
python manage.py migrate && \
python manage.py runserver 0.0.0.0:8001
Short Answer
No
Long Answer
When you run docker build the current directory and all of its contents (subdirectories and all) are copied into a staging area called the 'build context'. When you issue a COPY instruction in the Dockerfile, docker will copy from the staging area into a layer in the image's filesystem.
As you can see, this procludes copying files from directories outside the build context.
Workaround
Either download the files you want from their golden-source directly into the image during the build process (this is why you often see a lot of curl statements in Dockerfiles), or you can copy the files (dirs) you need into the build-tree and check them into source control as part of your project. Which method you choose is entirely dependent on the nature of your project and the files you need.
Notes
There are other workarounds documented for this, all of them without exception break the intent of 'portability' of your build. The only quality solutions are those documented here (though I'm happy to add to this list if I've missed any that preserve portability).
I am working on a project which has been deployed on compute engine on Debian 9 VM instance.
Here's are the steps I have performed:
Create Instance and ssh into
Install Python (3.6) directly by using these commands:
sudo apt-get update && sudo apt-get upgrade
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev
sudo apt-get install -y libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
sudo apt-get install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev
wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz
tar xvf Python-3.6.4.tgz
and some other build commands to ....
Then install virtualenv and activate it
Clone the django project from github and install the requirements
And run the gunicorn and the project is working fine.
But now, I want to set up the ssl for my Django projects, I have googled it but the tutorials are for nginx, I don't have setup nginx on my VM instance, so how can I set up SSL on my instance? I want to use Lets Encrypt.
I'm developing a Python tool which uses a sqlite3 virtual table with FTS5 (Full Text Search). I would like to know how to properly install from a tarball (or any other means) the needed requirements for my tool to work so I can pack them for portability.
Currently, I managed to install the latest release tarball of sqlite. However, when I execute:
python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
# or
python2 -c "import sqlite3; print(sqlite3.sqlite_version)"
I get 3.11.0, while sqlite3 --version returns: 3.22.0 2018-01-22 18:45:57 0c55d179733b46d8d0ba4d88e01a25e10677046ee3da1d5b1581e86726f2alt1
The system version sqlite3 3.22 does support FTS5, as I do pragma compile_options; and get:
COMPILER=gcc-5.4.0 20160609
ENABLE_DBSTAT_VTAB
ENABLE_FTS4
**ENABLE_FTS5**
ENABLE_JSON1
ENABLE_RTREE
ENABLE_STMTVTAB
ENABLE_UNKNOWN_SQL_FUNCTION
HAVE_ISNAN
THREADSAFE=1
But, the python version, using this script returns this:
[(u'ENABLE_COLUMN_METADATA',), (u'ENABLE_DBSTAT_VTAB',), (u'ENABLE_FTS3',), (u'ENABLE_FTS3_PARENTHESIS',), (u'ENABLE_JSON1',), (u'ENABLE_LOAD_EXTENSION',), (u'ENABLE_RTREE',), (u'ENABLE_UNLOCK_NOTIFY',), (u'ENABLE_UPDATE_DELETE_LIMIT',), (u'HAVE_ISNAN',), (u'LIKE_DOESNT_MATCH_BLOBS',), (u'MAX_SCHEMA_RETRY=25',), (u'OMIT_LOOKASIDE',), (u'SECURE_DELETE',), (u'SOUNDEX',), (u'SYSTEM_MALLOC',), (u'TEMP_STORE=1',), (u'THREADSAFE=1',)]
Hence, my questions are:
Is there any way I could make a linux portable package for my app
with sqlite3 FTS5 support in both python and linux system?
Is there any way to link the python module sqlite3 to an specific
sqlite3 path?
I tried all of this in an Ubuntu 16.04 LTS, but I would like to work as well on CentOS 7.
Thank you very much in advance.
More details about the installation from the tarball that I did:
wget "https://www.sqlite.org/src/tarball/sqlite.tar.gz?r=release" -O sqlite.tar.gz
tar -xzvf sqlite.tar.gz
cd sqlite
./configure --enable-fts5
make
sudo make install
The easy way is to use apsw (Another Python SQLite Wrapper). Its API is just a little different from sqlite3 and you can't just pip-install it (unless you're okay with outdated version), but the rest is good and you can have the most recent features of SQLite.
wget https://github.com/rogerbinns/apsw/releases/download/3.22.0-r1/apsw-3.22.0-r1.zip
unzip apsw-3.22.0-r1.zip
cd apsw-3.22.0-r1
python setup.py fetch --sqlite build --enable-all-extensions install
Then,
import apsw
apsw.Connection(':memory:').cursor().execute('pragma compile_options').fetchall()
Returns:
[('COMPILER=gcc-5.4.0 20160609',),
('ENABLE_API_ARMOR',),
('ENABLE_FTS3',),
('ENABLE_FTS3_PARENTHESIS',),
('ENABLE_FTS4',),
('ENABLE_FTS5',),
('ENABLE_ICU',),
('ENABLE_JSON1',),
('ENABLE_RBU',),
('ENABLE_RTREE',),
('ENABLE_STAT4',),
('THREADSAFE=1',)]
The hard way is to compile Python with custom SQLite. More detail in this article by Charles Leifer.
I think is a linking problem! I followed the same install steps with you and got the same results:
$ python ./test.py
[(u'ENABLE_COLUMN_METADATA',), (u'ENABLE_FTS3',), (u'ENABLE_RTREE',), (u'ENABLE_UNLOCK_NOTIFY',), (u'ENABLE_UPDATE_DELETE_LIMIT',), (u'MAX_SCHEMA_RETRY=25',), (u'OMIT_LOOKASIDE',), (u'SECURE_DELETE',), (u'SOUNDEX',), (u'SYSTEM_MALLOC',), (u'TEMP_STORE=1',), (u'THREADSAFE=1',)]
NO
However, when you install something by configure/make/make install on Linux, it usually goes in /usr/local/lib. To make sure that python links on runtime against the correct .so I used LD_LIBRARY_PATH. In this case I got:
$ LD_LIBRARY_PATH=/usr/local/lib python ./test.py
[(u'COMPILER=gcc-4.8.5',), (u'ENABLE_FTS5',), (u'HAVE_ISNAN',), (u'TEMP_STORE=1',), (u'THREADSAFE=1',)]
YES
Additionally, when installing libraries, you might have to update ldconfig. On my system (Ubuntu 14.04):
$ sudo ldconfig
$ python ./test.py
[(u'COMPILER=gcc-4.8.5',), (u'ENABLE_FTS5',), (u'HAVE_ISNAN',), (u'TEMP_STORE=1',), (u'THREADSAFE=1',)]
YES
Notice that using LD_LIBRARY_PATH is not needed any more and python links against the correct lib. For this to happen you will need to have /usr/local/lib folder in your ld.so.conf somewhere... for me this is in:
$ grep -ir local /etc/ld.so.conf.d/
/etc/ld.so.conf.d/libc.conf:/usr/local/lib
Thank you for your answers #urban and #saaj. I found your answers constructive.
The problem I see to #saaj answer is that it requires extra packages, specifically apsw package, which is not compatible with pypy, for example. I could not manage to make it work, but may be my fault.
I really like #urban answer. I did the process and got it working. I marked this answer as correct.
However I would like to add my own answer. Is quite aggressive but it worked for me. I created an Ubuntu docker with the following Dockerfile:
FROM ubuntu:16.04
RUN apt-get update -y
RUN DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true apt-get install -y apt-utils tzdata
RUN DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true dpkg-reconfigure tzdata
RUN echo "Europe/Berlin" > /etc/timezone
RUN dpkg-reconfigure -f noninteractive tzdata
RUN apt-get update -y
RUN apt-get install -y git build-essential sudo
Afterwards, inside the Ubuntu docker I did. In the process I remove sqlite3 and installed its dependencies, that I found in the following article. Afterwards I reinstalled python.
sudo apt-get update -y
echo "[ - Removing sqlite3... ]"
sudo apt-get remove -y sqlite3
sudo apt-get purge -y sqlite3
echo "[ - Installing sqlite3 dependencies... ]"
sudo apt-get install -y build-essential bzip2 git libbz2-dev libc6-dev libgdbm-dev libgeos-dev liblz-dev liblzma-dev libncurses5-dev libncursesw5-dev libreadline6 libreadline6-dev libsqlite3-dev libssl-dev lzma-dev python-dev python-pip python-software-properties python-virtualenv software-properties-common sqlite3 tcl tk-dev tk8.5-dev wget
echo "[ - Installing sqlite3... ]"
sudo wget "https://www.sqlite.org/src/tarball/sqlite.tar.gz?r=release" -O sqlite.tar.gz &> /dev/null
sudo tar -xzvf sqlite.tar.gz
cd sqlite
sudo ./configure --enable-fts5
sudo make
sudo make install
cd ..
echo "[ - Reinstalling python... ]"
sudo apt-get remove -y python python3 python-dev
sudo apt-get install -y --reinstall python2.7 python3 python-dev
sudo apt-get install -y build-essential bzip2 git libbz2-dev libc6-dev libgdbm-dev libgeos-dev liblz-dev liblzma-dev libncurses5-dev libncursesw5-dev libreadline6 libreadline6-dev libsqlite3-dev libssl-dev lzma-dev python-dev python-pip python-software-properties python-virtualenv software-properties-common sqlite3 tcl tk-dev tk8.5-dev wget