Develop Reference

Kafka producer not working in Docker container

I've created simple Kafka app that sends message to a topic. It works perfectly when I'm run it in local environment. But when I move it to Docker container it cannot connect to the broker. I think problem in container network settings but I cannot figure it out.
App code:
from kafka import KafkaProducer
producer = KafkaProducer(
bootstrap_servers='xxxxxxxxxx.mdb.yandexcloud.net:9091',
security_protocol="SASL_SSL",
sasl_mechanism="SCRAM-SHA-512",
sasl_plain_password='xxxxxxxxxx',
sasl_plain_username='xxxxxxxxxx',
ssl_cafile="YandexCA.crt",
api_version=(0,11,5))
producer.send('test_topic', b'test message')
producer.flush()
producer.close()
Dockerfile:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.10
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "app.py"]
So it's runs perfectly in terminal but fails in Docker. What can cause it?

So the problem was in the password. There was characters with escaping like:
ENV PASS=xxxxx\6xxxxx
And when set by env vars it worked correctly but when set in docker file it was escaped. So in Dockerfile I set it like that:
ENV PASS="xxxxx\6xxxxx"
And everything started working.

Missing environment variables on running python cronjob in Docker

I'm running a python script inside a docker container using crontab. Also, I set some environment variables (as database host, password, etc.) in .env file in the project's directory. If I run the script manually inside the container (python3 main.py) everything is working properly. But when the script is run by crontab the environment variables are not found (None).
I have the following setup:
Dockerfile
FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get -y install cron
RUN apt-get install -y python3-pip python-dev
WORKDIR /home/me/theservice
COPY . .
RUN chmod 0644 theservice-cron
RUN touch /var/log/theservice-cron.log
RUN chmod +x run.sh
ENTRYPOINT ./run.sh
run.sh
#!/bin/bash
crontab theservice-cron
cron -f
docker-compose.yml
version: '3.7'
services:
theservice:
build: .
env_file:
- ./.env
theservice-cron
HOME=/home/me/theservice
* * * * * python3 /home/me/theservice/main.py >> /var/log/theservice-cron.log 2>&1
#* * * * * cd /home/me/theservice && python3 main.py >> /var/log/theservice-cron.log 2>&1
I assumed that the cronjob is running in another directory and there the environment variables set in /home/me/theservice/.env are not accessible. So I tried to add HOME=/home/me/theservice line in the theservice-cron file or just to execute /home/me/theservice before running the script but it didn't help.
In the python script, I use os to access environment variables
import os
print(os.environ['db_host'])
How I can fix this problem?

I had similar problem.
I did fix it using the following:
CMD printenv > /etc/environment && cron && tail -f /var/log/theservice-cron.log
According to
https://askubuntu.com/questions/700107/why-do-variables-set-in-my-etc-environment-show-up-in-my-cron-environment, cron reads env vars from /etc/enviroment

For those fighting to get ENV variables from docker-compose into docker, simply have a shell script run at ENTRYPOINT in your Dockerfile, with
printenv > /etc/environment
again, the naming of "/etc/environment" is CRUCIAL !
And then in your crontab, have it call a shell script:
* * * * * bash -c "sh /var/www/html/cron_php.sh"
The scripts simply does :
#!/bin/bash
cd /var/www/html
php whatever.php
You will now have the docker-compose environment variables in your php cron application. It took me a full day to figure this out. Hope i save someone's trouble now !
UPDATE:
In Azure Docker (Web app) the mechanism doesn't seem to work. A small tweak is needed:
In the Dockerfile, in the ENTRYPOINT sh script, write a file (and CHMOD to execution rights chmod 770 ) /etc/environments.sh using this command:
eval $(printenv | awk -F= '{print "export " $1"=""""$2""" }' >> /etc/environments.sh)
Then, in your crontab shell where you execute php, do this:
#!/bin/bash
. /etc/environments.sh
php whatever.php
Notice the "." instead of source. Even though the Docker container is Linux using bash, source did not do the trick, the . did work.
Note: In my local Windows Docker the first solution, using /etc/envrionment worked fine. I was baffled to find out that on Azure the second fix was needed.

Python script inside docker creates empty file with CRON

I have a CRON job set like this :
0 0 * * * cd /home/path/to/script && sudo -u myuser ./thescript.sh
This script builds a docker image running a simple python app dumping on file a test validation report. When I run this script in a terminal everything works fine (the generated file goes in /home/myuser). Unfortunately, when I run the CRON job, the file is created, but empty. It must have something to do with the Root owning the CRON job, but I can't figure out how to get it done
Any clue ?

A sudo requires a TTY and cron doesn't run commands with a TTY. You need to run the cron for root user
This can be done using
sudo crontab -e
Then in cron don't use sudo
0 0 * * * cd /home/path/to/script && ./thescript.sh

Cron in Docker container parallel with Django

I am currently running a Django web application in a Docker compose setup on Rancher. Because I want the server to run Django management commands periodically I've setup a crontab for it.
* * * * * root /usr/local/bin/python /usr/src/app/manage.py updatesomething >> /usr/src/app/cron.log 2>&1
I am using the Dockerfile shown below and as you can see I've tried running the crontab standalone with CMD ["cron", "-f"]. This works fine and runs the command as it should. The idea however is that it can run parallel and trigger management commands on the web app. I've also verified that the crontab file is present.
The cron.log file remained empty for over 10 minutes so cron clearly is not doing its job here. Does anyone have a solution to running cron parallel in a python:3 container? Supervisor is not really an option as I have a Python 3 codebase. And I couldn't yet get Circus to work with the database in another container.
############################################################
# Dockerfile to run a Django-based web application
# Based on a Python 3 image
############################################################
# Set the base image to use to Python 3
FROM python:3
RUN apt-get update -qq && apt-get install -y -qq --force-yes cron
COPY ./docker/crontab/updatesomething /etc/cron.d/updatesomething
RUN chmod 0644 /etc/cron.d/updatesomething
RUN mkdir -p /usr/src/app /srv/logs
WORKDIR /usr/src/app
# Install dependencies
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
RUN cron
# Copy application files
COPY . /usr/src/app/
# Port to expose
EXPOSE 8000
# Copy entrypoint script into the image
COPY docker_entrypoint.sh /docker_entrypoint.sh
RUN chmod +x /docker_entrypoint.sh
CMD ["/docker_entrypoint.sh"]

If you want to run management commands periodically, have a look at Celery and use Celery beat. You can run tasks that call the management commands on specific times, the same as you would do with cron. Django has a way to call management commands from within the code. You can run Celery and Celery beat from your docker-compose setup.
from celery import shared_task
from django.core.management import call_command
#shared_task
def management_command_task():
call_command('my_command', 'foo', bar='baz')

Docker, dockercloud and Cron

I have a dockerfile which automates the building of an image.
I am using the docker cloud, connected to Digital Ocean as the server.
Within my dockerfile, I get the software I need, add the relevant GitHub repository containing the python scripts I wish to run. I then start the cron scheduler and add the script with appropriate times. For example:
The cron_files.txt file looks like this:
0 12 * * * /usr/bin/python /home/dir/run_check.py
0 15 * * * /usr/bin/python /home/dir/run_push.py
In my dockerfile, I do the following:
RUN service cron start
RUN service cron status
RUN crontab -u root cron_files.txt
In the log files, I can see that cron is succesfully started.
Edit, thanks to r0manarmy for this - How to run a cron job inside a docker container?
# Add crontab file in the cron directory
ADD crontab /etc/cron.d/hello-cron
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/hello-cron
# Create the log file to be able to run tail
RUN touch /var/log/cron.log
# Run the command on container startup
CMD cron && tail -f /var/log/cron.log
How do I edit the above to create the crontab file from the cron_files.txt rather than the above example?
I've tried ADD crontab cron_files.txt /etc/cron.d/feeds
But this returns:
ADD crontab cron_files.txt /etc/cron.d/feeds
lstat crontab: no such file or directory
Ps. I am using FROM debian:jessie

You probably want to set cron as the CMD:
In order to do this, just use the crond command on, say, alpine linux.
Take a look at this example for ideas:
https://gist.github.com/mhubig/a01276e17496e9fd6648cf426d9ceeec