Develop Reference

Install talib on docker

FROM python:3
USER root
RUN apt-get update
RUN apt-get -y install locales && \
localedef -f UTF-8 -i ja_JP ja_JP.UTF-8
RUN wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && \
tar -xvzf ta-lib-0.4.0-src.tar.gz && \
cd ta-lib/ && \
./configure --prefix=/usr && \
make && \
make install
RUN pip install TA-Lib
RUN rm -R ta-lib ta-lib-0.4.0-src.tar.gz
ENV LANG ja_JP.UTF-8
ENV LANGUAGE ja_JP:ja
ENV LC_ALL ja_JP.UTF-8
ENV TZ JST-9
ENV TERM xterm
ADD . /code
WORKDIR /code
RUN apt-get install -y vim less
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools
RUN pip install -r requirements.txt
version: "3"
services:
python3:
restart: always
build: .
container_name: "binancepython3"
working_dir: /root/
tty: true
volumes:
- ./opt:/root/opt
pandas
requests
ccxt == 1.81.77
I'm trying to install talib on docker, but I got an error like below, could you teach me how to solve it?
Is the problem caused by the environment? Should I use anaconda instead of python:3?
#7 3.276 configure: error: cannot guess build type; you must specify one
------
executor failed running [/bin/sh -c wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && tar -xvzf ta-lib-0.4.0-src.tar.gz && cd ta-lib/ && ./configure --prefix=/usr && make && make install]: exit code: 1
ERROR: Service 'python3' failed to build : Build failed

add:
./configure --build=aarch64-unknown-linux-gnu
more info: https://stackoverflow.com/a/68025766/1145929

How do I help Docker find Python 3.9 on my system?

I'm trying to build a rudimentary API and deploy it to a cloud environment with Docker. My API can be found in this directory here: https://github.com/n8feldman/cashman-flask-project. Basically, I've built a Flask API that can add "income" or "expense" items to an array through RESTful endpoints.
I downloaded Docker today and built the following Dockerfile in the root directory:
# Using lightweight alpine image
FROM python:3.6-alpine
# Installing packages
RUN apk update
RUN pip install --no-cache-dir pipenv
# Defining working directory and adding source code
WORKDIR /usr/src/app
COPY Pipfile Pipfile.lock bootstrap.sh ./
COPY cashman ./cashman
# Install API dependencies
RUN pipenv install
# Start app
EXPOSE 5000
ENTRYPOINT ["/usr/src/app/bootstrap.sh"]
I then run the command docker build -t cashman . from the root directory of my project and get the following output:
[+] Building 1.1s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.6-alpine 0.3s
=> [1/7] FROM docker.io/library/python:3.6-alpine#sha256:579978dec4602646fe1262f02b96371779bfb0294e92c91392707fa999c0c989 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 1.01kB 0.0s
=> CACHED [2/7] RUN apk update 0.0s
=> CACHED [3/7] RUN pip install --no-cache-dir pipenv 0.0s
=> CACHED [4/7] WORKDIR /usr/src/app 0.0s
=> CACHED [5/7] COPY Pipfile Pipfile.lock bootstrap.sh ./ 0.0s
=> CACHED [6/7] COPY cashman ./cashman 0.0s
=> ERROR [7/7] RUN pipenv install 0.7s
------
> [7/7] RUN pipenv install:
#11 0.607 Warning: Python 3.9 was not found on your system...
#11 0.608 Neither 'pyenv' nor 'asdf' could be found to install Python.
#11 0.608 You can specify specific versions of Python with:
#11 0.608 $ pipenv --python path/to/python
------
executor failed running [/bin/sh -c pipenv install]: exit code: 1
How can I decipher the error message at the bottom and fix the issue?

AppDynamics Serverless APM for Lambda. Auto instrument docker layer extension. Python app. Error = Extension.Crash

I'm trying to implement APM Serverless for AWS Lambda in a Python function. The function is deployed via a container image, so the extension is built as a layer in the Dockerfile.
Firstly, in case someone is trying to auto instrument this process; The extension must be unzipped into /opt/extensions/ not to /opt/ like suggested in the docs. Otherwise, the Lambda won't see the extension.
This is the Dockerfile:
FROM amazon/aws-cli:2.2.4 AS downloader
ARG version_number=10
ARG region=
ENV AWS_REGION=${region}
ENV VERSION_NUMBER=${version_number}
RUN yum install -y jq curl
WORKDIR /aws
RUN aws lambda get-layer-version-by-arn --arn arn:aws:lambda:$AWS_REGION:716333212585:layer:appdynamics-lambda-extension:$VERSION_NUMBER | jq -r '.Content.Location' | xargs curl -o extension.zip
# set base image (host OS)
FROM python:3.6.8-slim
ENV APPDYNAMICS_PYTHON_AUTOINSTRUMENT=true
# set the working directory for AppD
WORKDIR /opt
RUN apt-get clean \
&& apt-get -y update \
&& apt-get -y install python3-dev \
python3-psycopg2 \
&& apt-get -y install build-essential
COPY --from=downloader /aws/extension.zip .
RUN apt-get install -y unzip && unzip extension.zip -d /opt/extensions/ && rm -f extension.zip
# set the working directory in the container
WORKDIR /code
RUN pip install --upgrade pip \
&& pip install awslambdaric && pip install appdynamics-lambda-tracer
# copy the dependencies file to the working directory
COPY requirements.txt .
# install dependencies
RUN pip install -r requirements.txt
# copy the content of the local src directory to the working directory
COPY /src .
RUN chmod 644 $(find . -type f) \
&& chmod 755 $(find . -type d)
# command to run on container start
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "app.lambda_handler" ]
However, when executing the function, I get the following error:
AWS_Execution_Env is not supported, running lambda with default arguments.
EXTENSION Name: appdynamics-extension-script State: Started Events: []
Start RequestId <requestid>
End RequestId <requestid>
Error: exit code 0 Extension.Crash
Without more information.
When I try to implement the tracer manually, by installing appdynamics-lambda-tracer with pip, and importing the module I do see the logs in Cloudwatch from AppDynamics but they don't report to the controller.
Any idea what could be causing said crash?

Dockerfile build image error: Distribution contains no modules or packages for namespace package

I'm trying to build a docker image of a python project. My project has one dependency package, "my_package" which needs to be install first. It's a namespace package.
I'm having this error when building docker image of this python project. I can pretty much install everything with RUN pip install XYZ inside Dockerfile but I'm not sure what's the proper way to install local python namespace packages, here, my_package, properly.
I followed this exact same steps on a normal python package that is NOT A NAMESPACE PACKAGE and it works absolutely fine. Any pointers to install python namespace packages in Docker?
Dockerfile:
ARG BASE_IMAGE=ubuntu:18.04
FROM ${BASE_IMAGE} AS compile-image
ARG BASE_IMAGE=ubuntu:18.04
ENV PYTHONUNBUFFERED TRUE
RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt \
apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
ca-certificates \
g++ \
python3.8-dev \
python3.8-distutils \
python3.8-venv \
openjdk-11-jre-headless \
curl \
&& rm -rf /var/lib/apt/lists/* \
&& cd /tmp \
&& curl -O https://bootstrap.pypa.io/get-pip.py \
&& python3.8 get-pip.py
RUN python3.8 -m venv /home/venv
ENV PATH="/home/venv/bin:$PATH"
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 1
RUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3.8 1
COPY my_package/* /tmp/my_package/
WORKDIR /tmp/my_package/
RUN python setup.py install
Docker build command:
$ DOCKER_BUILDKIT=1 docker build --file Dockerfile .
Error:
[+] Building 0.6s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 907B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:18.04 0.1s
=> [internal] load build context 0.0s
=> => transferring context: 3.11kB 0.0s
=> [compile-image 1/8] FROM docker.io/library/ubuntu:18.04#sha256:4bc3ae6596938cb0d9e5ac51a1152ec9dcac2a1c50829c74abd9c4361e321b26 0.0s
=> CACHED [compile-image 2/8] RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --no-install 0.0s
=> CACHED [compile-image 3/8] RUN python3.8 -m venv /home/venv 0.0s
=> CACHED [compile-image 4/8] RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 1 0.0s
=> CACHED [compile-image 5/8] RUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3.8 1 0.0s
=> [compile-image 6/8] COPY my_package/* /tmp/my_package/ 0.0s
=> [compile-image 7/8] WORKDIR /tmp/my_package/ 0.0s
=> ERROR [compile-image 8/8] RUN python setup.py install 0.4s
------
> [compile-image 8/8] RUN python setup.py install:
#12 0.370 error in my_package setup command: Distribution contains no modules or packages for namespace package 'my_package'
------
executor failed running [/bin/sh -c python setup.py install]: exit code: 1
my_package structure:
my_package
├── [4.0K] my_package
│   └── [ 56] __init__.py
└── [ 162] setup.py
setup.py:
from setuptools import setup, find_packages
setup(
name='my_package',
version='0.0.1',
namespace_packages=['my_package'],
packages=find_packages()
)
__init__.py file:
__import__('pkg_resources').declare_namespace(__name__)

In your Dockerfile, you need to put :
COPY my_package/. /tmp/my_package/

Unable to locate package msodbcsql on Python image

This is the first time that I'm trying to use pyodbc to connect to an Azure SQL Database within a docker image. My Dockerfile looks like the below:
# the base image
FROM python:3
WORKDIR /usr/src/app
COPY requirements.txt ./
COPY music_trends.py ./
# install SQL Server drivers
RUN apt-get update
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql unixodbc-dev
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD [ "python", "./music_trends.py" ]
Which throws the error message:
E: Unable to locate package msodbcsql
The command '/bin/sh -c apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql unixodbc-dev' returned a non-zero code: 100
I have found resolutions for ubuntu:16.04 such as: https://github.com/Azure/azure-functions-docker/pull/45 and have also tried to run the msodbcsql.msi files from my Dockerfile.
Is there an equivalent fix for python:3?

python:3 is based on debian, so refer to microsoft doc:
You should install microsoft apt source, meanwhile change msodbcsql to msodbcsql17, example as next:
Dockerfile:
FROM python:3
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \
curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list && \
apt-get update && \
ACCEPT_EULA=Y apt-get install msodbcsql17 unixodbc-dev -y
UPDATE 2019-07-26:
I didn't notice official python:3 image update from debian 9 to debian 10 early this month, see this
From microsoft guide above, it seems currently they just package every dependency ok for next:
#Debian 8
curl https://packages.microsoft.com/config/debian/8/prod.list > /etc/apt/sources.list.d/mssql-release.list
#Debian 9
curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
Of course you can handle debian 10 dependency by yourself, such as libcrypto.so version issue etc, but I still suggest you just use python3 debian 9 version as microsoft did everything for you (PS: I think they will update in near future just because debian 10 release half month ago, I guess they need some time. BTW, https://packages.microsoft.com/config/debian/10/prod.list is there, but it does not have the package msodbcsql17 currently...)
So what I suggest easiest way for you is next, compared to the old Dockerfile, just change python:3 to python:3-stretch, and also install apt-transport-https which default not installed in debian 9, detail as follows:
Dockerfile:
FROM python:3-stretch
RUN apt-get update && \
apt-get install -y apt-transport-https && \
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \
curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list && \
apt-get update && \
ACCEPT_EULA=Y apt-get install msodbcsql17 unixodbc-dev -y
.so check:
root#91addb538736:/# ldd /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.3.so.1.1
linux-vdso.so.1 (0x00007ffd72bd0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4892696000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f489248e000)
libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2 (0x00007f4892273000)
libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007f4891e0d000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f4891b33000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f48918e8000)
libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007f489167f000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f489147a000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f48910f8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4890df4000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4890bdd000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f48909c0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4890621000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4892ca1000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f48903ee000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f48901ea000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f488ffde000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f488fdda000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f488fbc3000)