I have a cronjob that runs every 5th minute which runs a bash script, which then runs a python file. All of this is running in a docker container:
crontab:
*/5 * * * * root ./rates/cmd.sh
cmd.sh:
#!/usr/bin/env bash
result=$(/usr/local/bin/python3 __main__.py --env $DB_ENVIRONMENT)
echo "$result" >> logs/"$(date +"%Y-%m-%d")".log;
main.py:
parser = argparse.ArgumentParser()
parser.add_argument("-e", "--env", dest="env", help="environments (choose 'dev' or 'live'); default is dev")
options = parser.parse_args()
print(options.env)
Docker Commands:
docker build -t rates .
docker run -d -e DB_ENVIRONMENT=dev rates
I am "sshing" into the docker container, and can verify that the DB_ENVIRONMENT variable has been set properly using "echo $DB_ENVIRONMENT".
The python script does not execute the print statement portion of the code, and I am assuming it fails on the parse_args() line. Nothing is outputted to the log file, when I am expecting the log to contain "dev".
When I run the bash script manually with ./cmd.sh or /usr/local/bin/python3 __main__.py --env $DB_ENVIRONMENT, the log file has "dev" in there / I can see output.
What is happening here?
Related
I have to run a docker and then a command inside the workdir using a python script.
i'm triyng to do it as follows:
command = ['gnome-terminal', '-e', "bash -c 'sudo /home/mpark/Escriptori/SRTConverter/shell_docker.sh; echo b; exec $SHELL'"]
p = subprocess.Popen(command)
where 'sudo /home/mpark/Escriptori/SRTConverter/shell_docker.sh' is a shell script with the docker run with root privileges
the fisrt command 'sudo /home/mpark/Escriptori/SRTConverter/shell_docker.sh' works good, but the second one 'echo b' that has to run inside the container doesn't work..
Thank you!
I'm using a python script for send websocket notification,
as suggested here.
The script is _wsdump.py and I have a script script.sh that is:
#!/bin/sh
set -o allexport
. /root/.env set
env
python3 /utils/_wsdump.py "wss://mywebsocketserver:3000/message" -t "message" &
If I try to dockerizing this script with this Dockerfile:
FROM python:3.8-slim-buster
RUN set -xe \
pip install --upgrade pip wheel && \
pip3 install websocket-client
ENV TZ="Europe/Rome"
ADD utils/_wsdump.py /utils/_wsdump.py
ADD .env /root/.env
ADD script.sh /
ENTRYPOINT ["./script.sh"]
CMD []
I have a strange behaviour:
if I execute docker run -it --entrypoint=/bin/bash mycontainer and after that I call the script.sh everything works fine and I receive the notification.
if I run mycontainer with docker run mycontainer I see no errors but the notification doesn't arrive.
What could be the cause?
Your script doesn't launch a long-running process; it tries to start something in the background and then completes. Since the script completes, and it's the container's ENTRYPOINT, the container exits as well.
The easy fix is to remove the & from the end of the last line of the script to cause the Python process to run in the foreground, and the container will stay alive until the process completes.
There's a more general pattern of an entrypoint wrapper script that I'd recommend adopting here. If you look at your script, it does two things: (1) set up the environment, then (2) run the actual main container command. I'd suggest using the Docker CMD for that actual command
# end of Dockerfile
ENTRYPOINT ["./script.sh"]
CMD python3 /utils/_wsdump.py "wss://mywebsocketserver:3000/message" -t "message"
You can end the entrypoint script with the magic line exec "$#" to run the CMD as the actual main container process. (Technically, it replaces the current shell script with a command constructed by replaying the command-line arguments; in a Docker context the CMD is passed as arguments to the ENTRYPOINT.)
#!/bin/sh
# script.sh
# set up the environment
. /root/.env set
# run the main container command
exec "$#"
With this use you can debug the container setup by replacing the command part (only), like
docker run --rm your-image env
to print out its environment. The alternate command env will replace the Dockerfile CMD but the ENTRYPOINT will remain in place.
You install script.sh to the root dir /, but your ENTRYPOINT is defined to run the relative path ./script.sh.
Try changing ENTRYPOINT to reference the absolute path /script.sh instead.
I edit the crontab in Ubuntu 18.04:
crontab -e
50 21 * * * /data/min/query/sync.sh >> logs/tag.log
Then I saved it and exited. I did this at 21:41pm, and I suppose it will execute it at 21:50pm, but it didn't effect. I checked the logs/tag.log, but nothing occurs.
The content of my sync.sh:
#!/bin/bash
cd /data/min/query
echo 'sync tags now ...'
source venv/bin/activate
python utils/tag_counter.py
echo 'updating resource files are done!!!'
I make the script executable by:
chmod +x sync.sh
So that sync.sh can be executed directly. The script does work when executed at the command line, but not working in the cronjobs. This is the first time I use cronjob. Is there anything I am missing in setting up the cronjob?
I'm running a python script inside a docker container using crontab. Also, I set some environment variables (as database host, password, etc.) in .env file in the project's directory. If I run the script manually inside the container (python3 main.py) everything is working properly. But when the script is run by crontab the environment variables are not found (None).
I have the following setup:
Dockerfile
FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get -y install cron
RUN apt-get install -y python3-pip python-dev
WORKDIR /home/me/theservice
COPY . .
RUN chmod 0644 theservice-cron
RUN touch /var/log/theservice-cron.log
RUN chmod +x run.sh
ENTRYPOINT ./run.sh
run.sh
#!/bin/bash
crontab theservice-cron
cron -f
docker-compose.yml
version: '3.7'
services:
theservice:
build: .
env_file:
- ./.env
theservice-cron
HOME=/home/me/theservice
* * * * * python3 /home/me/theservice/main.py >> /var/log/theservice-cron.log 2>&1
#* * * * * cd /home/me/theservice && python3 main.py >> /var/log/theservice-cron.log 2>&1
I assumed that the cronjob is running in another directory and there the environment variables set in /home/me/theservice/.env are not accessible. So I tried to add HOME=/home/me/theservice line in the theservice-cron file or just to execute /home/me/theservice before running the script but it didn't help.
In the python script, I use os to access environment variables
import os
print(os.environ['db_host'])
How I can fix this problem?
I had similar problem.
I did fix it using the following:
CMD printenv > /etc/environment && cron && tail -f /var/log/theservice-cron.log
According to
https://askubuntu.com/questions/700107/why-do-variables-set-in-my-etc-environment-show-up-in-my-cron-environment, cron reads env vars from /etc/enviroment
For those fighting to get ENV variables from docker-compose into docker, simply have a shell script run at ENTRYPOINT in your Dockerfile, with
printenv > /etc/environment
again, the naming of "/etc/environment" is CRUCIAL !
And then in your crontab, have it call a shell script:
* * * * * bash -c "sh /var/www/html/cron_php.sh"
The scripts simply does :
#!/bin/bash
cd /var/www/html
php whatever.php
You will now have the docker-compose environment variables in your php cron application. It took me a full day to figure this out. Hope i save someone's trouble now !
UPDATE:
In Azure Docker (Web app) the mechanism doesn't seem to work. A small tweak is needed:
In the Dockerfile, in the ENTRYPOINT sh script, write a file (and CHMOD to execution rights chmod 770 ) /etc/environments.sh using this command:
eval $(printenv | awk -F= '{print "export " $1"=""""$2""" }' >> /etc/environments.sh)
Then, in your crontab shell where you execute php, do this:
#!/bin/bash
. /etc/environments.sh
php whatever.php
Notice the "." instead of source. Even though the Docker container is Linux using bash, source did not do the trick, the . did work.
Note: In my local Windows Docker the first solution, using /etc/envrionment worked fine. I was baffled to find out that on Azure the second fix was needed.
Using ubuntu's 16.04 crontab and #reboot to run python3 script. The script runs properly on reboot as I see the logged output. However, my script's os.system command is not running. It runs fine if ran outside of crontab. My scripts are all executable.
crontab -l output:
SHELL=/bin/bash
#reboot nohup /usr/bin/python3 -u /home/path/scheduler.py >> /path/log.out &
scheduler.py code:
#...(check if web server is running...if not restart)
os.system('nohup /usr/bin/python3 -u /path/webserver/main.py &')
print('this function ran')
When I logged the output of the os.system command , there was no output.
As a side note, I am running python schedule commands to check the general health of a webserver. crontab doesn't seem to be the right tool for this so I just use crontab to start my python scheduler on reboot.
I am using flask as the webserver, and would use gunicorn and systemctrl if I could get it to work... but it didn't so this is my workaround.
The point is that, the command called by os.system is not in default path.
For example, tcpdump is not in /usr/bin/.
So, you can solve the problem by adding the full path of the command.
I was facing the same issue when we try to run python script directly in crontab it just by passes the os.system() commands.
Make launcher.sh:
#!bin/bash
cd /home/pi/
sudo python example.py
Then, make your script executable:
chmod 755 launcher.sh
And at last, add your script to crontab:
crontab -e
and add this line at the end:
#reboot sh /home/pi/launcher.sh
(I set the program to run at each reboot)