Develop Reference

DOCKER_BUILDKIT - passing a token secret during build time from github actions

I have a pip installable package that was published using twine to Azure DevOps artifact.
In my build image I need to download that package and install it with pip. So I need to
authenticate against azure artifacts to get it. So I am using artifacts-keyring to do so
The pip installable URL is something like this:
https://<token>#pkgs.dev.azure.com/<org>/<project>/_packaging/<feed>/pypi/simple/ --no-cache-dir <package-name>
I am trying to use Docker BuildKit to pass the token during build time.
I am mounting the secret and using it by getting its value with cat command substitution:
# syntax = docker/dockerfile:1.2
ENV ARTIFACTS_KEYRING_NONINTERACTIVE_MODE=true
RUN --mount=type=secret,id=azdevopstoken,dst=/run/secrets/azdevopstoken \
pip install --upgrade pip && \
pip install pyyaml numpy lxml artifacts-keyring && \
echo pip install -i https://"$(cat /run/secrets/azdevopstoken)"#pkgs.dev.azure.com/org>/<project>/_packaging/feed/pypi/simple/ --no-cache-dir package-name
and it works locally from my workstation when I run:
(My src file where the token is in plain text is azdevopstoken.txt within my local directory structure project)
DOCKER_BUILDKIT=1 docker image build --secret id=azdevopstoken,src=./azdevopstoken.txt --progress plain . -t my-image:tag
Now I am running this build command from GitHub actions pipeline
And I got this output:
Scould not parse secrets: [id=azdevopstoken,src=./azdevopstoken.txt]: failed to stat ./azdevopstoken.txt: stat ./azdevopstoken.txt: no such file or directory
Error: Process completed with exit code 1.
This is expected from me, since I am not uploading azdevopstoken.txt file, because I don’t want to have it in my repository, since there is my token in plain text.
Reading carefully here, I see there is a workaround to encrypt secrets, perhaps this could be a good solution to implement buildkit from my github actions pipeline, but I think this is an additional step in my workflow, so I am not sure whether follow this option or not.
Foremost because I already passing the secret in the old way by using the --build-arg flag during build time of this way:
ARG AZ_DEVOPS_TOKEN
ENV AZ_DEVOPS_TOKEN=$AZ_DEVOPS_TOKEN
RUN pip install --upgrade pip && \
pip install pyyaml numpy lxml artifacts-keyring && \
pip install -i https://$AZ_DEVOPS_TOKEN#pkgs.dev.azure.com/ORG/PROJECT/_packaging/feed/pypi/simple/ --no-cache-dir PACKAGE-NAME
Being my docker build command from GitHub actions of this way:
docker image build --build-arg AZ_DEVOPS_TOKEN="${{ secrets.AZ_DEVOPS_TOKEN }}" . -t containerregistry.azurecr.io/my-image:preview
It works perfect, the thing is, I heard --build-arg is not a safe solution to pass sensitive information. Despite that I ran docker history after this command, and I couldn't see the secret exposed or something similar.
> docker history af-fem-uploader:preview-buildarg
IMAGE CREATED CREATED BY SIZE COMMENT
2eff105408c9 34 seconds ago RUN /bin/sh -c pip install pytest && pytest … 5MB buildkit.dockerfile.v0
<missing> 38 seconds ago WORKDIR /home/site/wwwroot/HttpUploadTrigger/ 0B buildkit.dockerfile.v0
<missing> 38 seconds ago ADD . /home/site/wwwroot # buildkit 1.9MB buildkit.dockerfile.v0
So what is the benefit to pass the secrets via BUILDKIT in order to don’t expose them if I have to upload the file which contains the secret to my repository?
Perhaps I am missing something.

Python Code running in Docker not working with NGROK

rather new to coding in general and especially new to docker & python so please excuse any blatant mistakes. I have been struggling with a specific issue for weeks now and maybe someone here can help me.
A webservice we'd like to use needs to be connected to a PostgreSQL Database via a Python connector. This connector basically compares data input on the webservice with the data in the DB and matches variables.
https://github.com/rossumai/elis-connector-example-python
Right now I am running the databse in AWS and I have succesfully uploaded sample data on it. Also I have run the connector.py code in Docker via the localhost:5000 port. Now I want to connect all the parts and I am running Ngrok to tunnel my localhost:5000 and I tell the webservice to connect to the https address ngrok is giving me.
Unfortunately I am receiving the "Failed to complete tunnel connection: The connection was successfully tunneled but the client failed to establish a connection to localhost:5000"
Is this a port issue?
This is my Dockerfile:
FROM python:3.7
WORKDIR /src/connector.py
COPY requirements.txt ./
RUN apt-get update && apt-get install -y libpq-dev gcc
RUN pip3 install psycopg2~=2.7.7
RUN apt-get autoremove -y gcc
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD [ "python", "./connector.py" ]
Any help, or what else can I add to make my problem a bit more comprehensible?
BR & thanks for any help

How to run SSL_library_init() from Python 3.7 docker image

Up until recently I've been using openssl library within python:3.6.6-jessie docker image and thing worked as intented.
I'm using very basic Dockerfile configuration to install all necessary dependencies:
FROM python:3.6.6-jessie
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
WORKDIR /code
RUN apt-get -qq update
RUN apt-get install openssl
RUN apt-get upgrade -y openssl
ADD requirements.txt /code/
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
And access and initialize the library itself with these 2 lines:
openssl = cdll.LoadLibrary("libssl.so")
openssl.SSL_library_init()
Things were working great with this approach.
This week I was doing upgrade of python and libraries and as result I switched to newer docker image:
FROM python:3.7.5
...
This immediatelly caused openssl to stop working because of this exception:
AttributeError: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: undefined symbol: SSL_library_init
From this error I can understand that libssl no longer provides SSL_library_init method (or so it seems to be) which is rather weird issue because the initializer name in openssl documentation is the same.
I also tried to resolve this using -stretch and -buster distributions but the issue remains.
What is the correct approach to run SSL_library_init in those newer distributions? Maybe some additional dockerfile configuration is required?

I think you need to install libssl1.0-dev
RUN apt-get install -y libssl1.0-dev

CoreOS build docker image (CP100A Training)

I'm currently working on the "Google cloud platform fundamentals" labs and I'm running into issues.
Each time I have to use a CoreOS instance to spin up a docker instance there is an error I get.
For example: in the Cloud SQL lab, at some point I have to build a docker image of the folder I just cloned from a git repo using the command:
docker build -t cp100/cloudsql-python cp100-cloud-sql-python
which gives me a wall of text that ends with an error :
Downloading/unpacking flask
Cannot fetch index base URL http://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement flask
No distributions at all found for flask
Storing complete log in /root/.pip/pip.log`
The thing Is, there is no "/root/.pip/pip.log" file.
So here are my questions :
Are the tutorials outdated, and if yes, where can I find the up-to-date tutorials?
Why does it happen? I think It is because pip or Python or both are not installed but shouldn't the command docker build take the installation in charge?
How can I fix it?
the cp100-cloud-sql-python file is available at https://github.com/GoogleCloudPlatformTraining/cp100-cloud-sql-python.git
Thanks for your answers.

Ok I found the answers by myself:
So the reason it doesn't work is that pip (and easy install) use HTTP and pypi.python.org requires HTTPS, the issue is further documented here :
https://bugzilla.redhat.com/show_bug.cgi?id=1510444
So in order to fix it I modified the Dockerfile inside the app from
FROM google/debian:wheezy
MAINTAINER Sharif Salah <sharif.salah+docker#gmail.com>
RUN apt-get update && \
apt-get install -y python-dev python-pip python-mysqldb && \
pip install flask
ADD app /app
EXPOSE 80
CMD [ "python", "/app/app.py" ]
to
FROM google/debian:wheezy
MAINTAINER Sharif Salah <sharif.salah+docker#gmail.com>
RUN apt-get update && \
apt-get install -y python-dev python-setuptools python-mysqldb && \
easy_install -i https://pypi.python.org/simple flask
ADD app /app
EXPOSE 80
CMD [ "python", "/app/app.py" ]
which will force easy_install to use the address specified after the -i.
It worked in my case but as documented on Bugzilla, it may not work for everything.
I hope this will help someone

"ssl module in Python is not available"

still advancing in my tutorial to learn python, I was told to do
sudo -H pip install requests
I get the following :
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting requests
Could not fetch URL https://pypi.python.org/simple/requests/: There was a problem confirming the ssl certificate: Can't connect to HTTPS URL because the SSL module is not available. - skipping
Could not find a version that satisfies the requirement requests (from versions: )
No matching distribution found for requests
would someone know what I need to do to fix that once for all ?
thanks so much in advance

I use Linux distros and face the problem since the new installation of Python 3.6.
I tried a couple of solution and finally solved the problem. The steps I followed are as below.
On Debian like distros
sudo apt-get install build-essential checkinstall libreadline-gplv2-dev ibncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
Change the directory to the Python3.6 location
cd /path/to/Python3.6/Module/
In the module directory, open the Setup file with your preferred text editor
vi Setup
Search for SSL and uncomment the related lines.
ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
Save the file and on the root folder of your Python package, run the following command.
make
sudo make install
And finally, run the pip3 to install your required module(s).

Running ./configure with --enable-optimizations did the trick.
Here are the steps that worked for me on a Ubuntu 16.04 LTS box-
The lines related to ssl in the Modules/Setup* files are commented
cd to the directory where you have the Python tar extracted
cd /../../../Python-3.7.4/
Run configure with optimizations enabled
./configure --enable-optimizations
Run make and then make install
make
make install

I encountered this problem running pip on Powershell on Windows, using the Anaconda distribution. I was running it inside VSCode, but I don't think it makes much difference.
A quick turnaround for me was installing what I needed using the Anaconda prompt, which works fine.

I encountered the problem when installing 3.8.11. I tried the above posts but they didn't work.
This is how I finally made it compile:
1) Install openssl 1.1.1g;
2) untar the tarball;
3) enter the source directory;
4) run ./config --prefix=/usr/local/openssl_1.1 --openssldir=/usr/local/openssl_1.1
to install to /usr/local/openssl_1.1.
make & make install.
You can customize the directory.
5)Download python 3.8 source tarball;
6)untar the tarball. Enter the source/Modules;
7) edit the file Setup;
8) find the lines:
SSL=/usr/local/ssl
ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
change SSL=/usr/local/ssl to SSL=/usr/local/openssl_1.1
9) Switch to the source directory. run
./configure --prefix=/usr/local/python3.8 --with-openssl=/usr/local/openssl_1.1
10) export LD_LIBRARY_PATH=/usr/local/openssl_1.1/lib:$LD_LIBRARY_PATH
11) make & make install. If you need sudo, also export LD_LIBRARY_PATH in sudo

This command worked very well for me.
cd Python-3.6.2
./configure --with-ssl
make
sudo make install

I encountered the same problem in Windows 10.
What I did was:
Step 1: go to https://pypi.python.org/simple/requests and download the latest version (e.g., requests-2.21.0.tar.gz).
Step 2: unzip the downloaded file into a folder (e.g., c:\temp\requests-2.21.0). You can use 7zip for that purpose.
Step 3: pip install c:\temp\requests-2.21.0
Note: pip can also install a local folder.
It worked for me.