I would like to make a Flask API running into docker with a conda environment.
It seems that I can install the conda environment from the .yml file.
But I can't run the app when I do docker run.
I just have errors about files that do not exist
exec source activate flask_env && python app.py failed: No such file or directory
The flask API is based on a simple example:
from flask import Flask, jsonify
app = Flask(__name__)
#app.route('/', methods=['GET'])
def hello_world():
return jsonify({'message': 'Hello World'})
#app.route('/test', methods=['GET'])
def test():
return jsonify({'test': 'test'})
if __name__ == "__main__":
app.run(debug=True) # remember to set debug to False
The Dockerfile is:
FROM continuumio/miniconda3:latest
WORKDIR /app
# Install myapp requirements
COPY environment.yml /app/environment.yml
RUN conda config --add channels conda-forge \
&& conda env create -n myapp -f environment.yml \
&& rm -rf /opt/conda/pkgs/*
# Copy all files after to avoid rebuild the conda env each time
COPY . /app/
# activate the myapp environment
ENV PATH /opt/conda/envs/myapp/bin:$PATH
# Launch the API
CMD [ "source activate flask_env && python app.py" ]
And the environment file is:
name: myapp
channels:
- conda-forge
- defaults
dependencies:
- flask=1.0.2
- python=3.7.3
I tried a lot of thing but I can't make it works. Did I miss something ?
Thank you
See this:
The CMD instruction has three forms:
CMD ["executable","param1","param2"] (exec form, this is the preferred form)
CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
CMD command param1 param2 (shell form)
Here, you CMD is used as parameters of ENTRYPOINT(see this), so you have to use next format:
CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
But, your command have && in it, so you have to enable shell in json format. So for your case, this could be next, FYI:
CMD ["bash", "-c", "source activate flask_env && python app.py"]
Related
I've created simple Kafka app that sends message to a topic. It works perfectly when I'm run it in local environment. But when I move it to Docker container it cannot connect to the broker. I think problem in container network settings but I cannot figure it out.
App code:
from kafka import KafkaProducer
producer = KafkaProducer(
bootstrap_servers='xxxxxxxxxx.mdb.yandexcloud.net:9091',
security_protocol="SASL_SSL",
sasl_mechanism="SCRAM-SHA-512",
sasl_plain_password='xxxxxxxxxx',
sasl_plain_username='xxxxxxxxxx',
ssl_cafile="YandexCA.crt",
api_version=(0,11,5))
producer.send('test_topic', b'test message')
producer.flush()
producer.close()
Dockerfile:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.10
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "app.py"]
So it's runs perfectly in terminal but fails in Docker. What can cause it?
So the problem was in the password. There was characters with escaping like:
ENV PASS=xxxxx\6xxxxx
And when set by env vars it worked correctly but when set in docker file it was escaped. So in Dockerfile I set it like that:
ENV PASS="xxxxx\6xxxxx"
And everything started working.
I am trying to deploy to cloud run using it's --command flag option (see https://cloud.google.com/sdk/gcloud/reference/run/deploy) but it fails anytime I try it. I do not know if I am not understanding how to use it or if it is something happening in google cloud's side.
My Dockerfile looks like the following:
FROM python:3.10-slim
ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
RUN pip install --no-cache-dir -r requirements.txt
ENTRYPOINT ["python"]
CMD ["main.py"]
And I am deploying it with:
gcloud run deploy \
$SERVICE \
--image $IMAGE \
--allow-unauthenticated \
--memory $MEMORY \
--concurrency $CONCURRENCY \
--cpu $CPU \
--platform managed \
--region $REGION \
--command "main2.py"
The logs are as follows:
X Deploying... Internal error occurred while performing container health check.
- Creating Revision...
. Routing traffic...
✓ Setting IAM Policy...
Aborted by user.
ERROR: (gcloud.run.deploy) Aborted by user.
Y tried also using only CMD in the Dockerfile (replace 2 last lines with CMD python main.py) and using --command "python main2.py" without success. I want to use the same Docker image but being able to deploy to run main.py (as default in Dockerfile) or main2.py
Note that if the --command flag is omitted it is deployed successfully and the app works.
The test code is at https://github.com/charlielito/cloud-run-test
The python code is just a dummy flask server. The main2.py is the same for testing purposes, just changed the response string.
import os
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello_world():
name = os.environ.get("NAME", "World")
return "Hello {}!".format(name)
if __name__ == "__main__":
app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
I'm running a python script inside a docker container using crontab. Also, I set some environment variables (as database host, password, etc.) in .env file in the project's directory. If I run the script manually inside the container (python3 main.py) everything is working properly. But when the script is run by crontab the environment variables are not found (None).
I have the following setup:
Dockerfile
FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get -y install cron
RUN apt-get install -y python3-pip python-dev
WORKDIR /home/me/theservice
COPY . .
RUN chmod 0644 theservice-cron
RUN touch /var/log/theservice-cron.log
RUN chmod +x run.sh
ENTRYPOINT ./run.sh
run.sh
#!/bin/bash
crontab theservice-cron
cron -f
docker-compose.yml
version: '3.7'
services:
theservice:
build: .
env_file:
- ./.env
theservice-cron
HOME=/home/me/theservice
* * * * * python3 /home/me/theservice/main.py >> /var/log/theservice-cron.log 2>&1
#* * * * * cd /home/me/theservice && python3 main.py >> /var/log/theservice-cron.log 2>&1
I assumed that the cronjob is running in another directory and there the environment variables set in /home/me/theservice/.env are not accessible. So I tried to add HOME=/home/me/theservice line in the theservice-cron file or just to execute /home/me/theservice before running the script but it didn't help.
In the python script, I use os to access environment variables
import os
print(os.environ['db_host'])
How I can fix this problem?
I had similar problem.
I did fix it using the following:
CMD printenv > /etc/environment && cron && tail -f /var/log/theservice-cron.log
According to
https://askubuntu.com/questions/700107/why-do-variables-set-in-my-etc-environment-show-up-in-my-cron-environment, cron reads env vars from /etc/enviroment
For those fighting to get ENV variables from docker-compose into docker, simply have a shell script run at ENTRYPOINT in your Dockerfile, with
printenv > /etc/environment
again, the naming of "/etc/environment" is CRUCIAL !
And then in your crontab, have it call a shell script:
* * * * * bash -c "sh /var/www/html/cron_php.sh"
The scripts simply does :
#!/bin/bash
cd /var/www/html
php whatever.php
You will now have the docker-compose environment variables in your php cron application. It took me a full day to figure this out. Hope i save someone's trouble now !
UPDATE:
In Azure Docker (Web app) the mechanism doesn't seem to work. A small tweak is needed:
In the Dockerfile, in the ENTRYPOINT sh script, write a file (and CHMOD to execution rights chmod 770 ) /etc/environments.sh using this command:
eval $(printenv | awk -F= '{print "export " $1"=""""$2""" }' >> /etc/environments.sh)
Then, in your crontab shell where you execute php, do this:
#!/bin/bash
. /etc/environments.sh
php whatever.php
Notice the "." instead of source. Even though the Docker container is Linux using bash, source did not do the trick, the . did work.
Note: In my local Windows Docker the first solution, using /etc/envrionment worked fine. I was baffled to find out that on Azure the second fix was needed.
I am building Python container for the first time using VS Code and WSL2. Here is my sample Python code. It runs fine in VS interactive mode because it is picking up my default AWS credentials.
import boto3
s3BucketName = 'test-log-files'
s3 = boto3.resource('s3')
def s3move():
try:
s3.Object(s3BucketName, "destination/Changes.xlsx").copy_from(CopySource=(s3BucketName + "/source/Changes.xlsx"))
s3.Object(s3BucketName,"source/Changes.xlsx").delete()
print("Transfer Complete")
except:
print("Transfer failed")
if __name__ == "__main__":
s3move()
The Dockerfile built by VS Code:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim-buster
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
RUN pip install boto3
WORKDIR /app
COPY . /app
# Switching to a non-root user, please refer to https://aka.ms/vscode-docker-python-user-rights
RUN useradd appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "S3MoveFiles/S3MoveFiles.py"]
I would like to test this using docker container and seems like I have to pass the AWS credentials to the container. While there are other ways and probably more secure ways, I wanted to test the method by mounting the volume in a docker command as an argument.
docker run -v ~/.aws/credentials:/appuser/home/.aws/credentials:ro image_id
I get the "Transfer failed" message in the Terminal window in VS Code. What am I doing wrong here? Checked several articles but couldn't find any hints. I am not logged in as root.
Being new to python & docker, I created a small flask app (test.py) which has two hardcoded values:
username = "test"
password = "12345"
I'm able to create a Docker image and run a container from the following Dockerfile:
FROM python:3.6
RUN mkdir /code
WORKDIR /code
ADD . /code/
RUN pip install -r requirements.txt
EXPOSE 5000
CMD ["python", "/code/test.py"]`
How can I create a ENV variable for username & password and pass dynamic values while running containers?
Within your python code you can read env variables like:
import os
username = os.environ['MY_USER']
password = os.environ['MY_PASS']
print("Running with user: %s" % username)
Then when you run your container you can set these variables:
docker run -e MY_USER=test -e MY_PASS=12345 ... <image-name> ...
This will set the env variable within the container and these will be later read by the python script (test.py)
More info on os.environ and docker env
In your Python code you can do something like this:
# USERNAME = os.getenv('NAME_OF_ENV_VARIABLE','default_value_if_no_env_var_is_set')
USERNAME = os.getenv('USERNAME', 'test')
Then you can create a docker-compose.yml file to run your dockerfile with:
version: '2'
services:
python-container:
image: python-image:latest
environment:
- USERNAME=test
- PASSWORD=12345
You will run the compose file with:
$ docker-compose up
All you need to remember is to build your dockerfile that you mentioned in your question with:
$ docker build -t python-image .
Let me know if that helps. I hope that answers your question.
FROM python:3
MAINTAINER <abc#test.com>
ENV username=test
password=12345
RUN mkdir /dir/name
RUN cd /dir/name && pip3 install -r requirements.txt
WORKDIR /dir/name
ENTRYPOINT ["/usr/local/bin/python", "./test.py"]
I split my docker-compose into docker-compose.yml (base), docker-compose.dev.yml, etc., then I had this issue.
I solved it by specifying the .env file explicitly in the base:
web:
env_file:
- .env
Not sure why, according to the docs it should just work if there's an .env file.