Develop Reference

Why a dockerized script have a different behaviour when I docker run or I docker execute it?

I'm using a python script for send websocket notification,
as suggested here.
The script is _wsdump.py and I have a script script.sh that is:
#!/bin/sh
set -o allexport
. /root/.env set
env
python3 /utils/_wsdump.py "wss://mywebsocketserver:3000/message" -t "message" &
If I try to dockerizing this script with this Dockerfile:
FROM python:3.8-slim-buster
RUN set -xe \
pip install --upgrade pip wheel && \
pip3 install websocket-client
ENV TZ="Europe/Rome"
ADD utils/_wsdump.py /utils/_wsdump.py
ADD .env /root/.env
ADD script.sh /
ENTRYPOINT ["./script.sh"]
CMD []
I have a strange behaviour:
if I execute docker run -it --entrypoint=/bin/bash mycontainer and after that I call the script.sh everything works fine and I receive the notification.
if I run mycontainer with docker run mycontainer I see no errors but the notification doesn't arrive.
What could be the cause?

Your script doesn't launch a long-running process; it tries to start something in the background and then completes. Since the script completes, and it's the container's ENTRYPOINT, the container exits as well.
The easy fix is to remove the & from the end of the last line of the script to cause the Python process to run in the foreground, and the container will stay alive until the process completes.
There's a more general pattern of an entrypoint wrapper script that I'd recommend adopting here. If you look at your script, it does two things: (1) set up the environment, then (2) run the actual main container command. I'd suggest using the Docker CMD for that actual command
# end of Dockerfile
ENTRYPOINT ["./script.sh"]
CMD python3 /utils/_wsdump.py "wss://mywebsocketserver:3000/message" -t "message"
You can end the entrypoint script with the magic line exec "$#" to run the CMD as the actual main container process. (Technically, it replaces the current shell script with a command constructed by replaying the command-line arguments; in a Docker context the CMD is passed as arguments to the ENTRYPOINT.)
#!/bin/sh
# script.sh
# set up the environment
. /root/.env set
# run the main container command
exec "$#"
With this use you can debug the container setup by replacing the command part (only), like
docker run --rm your-image env
to print out its environment. The alternate command env will replace the Dockerfile CMD but the ENTRYPOINT will remain in place.

You install script.sh to the root dir /, but your ENTRYPOINT is defined to run the relative path ./script.sh.
Try changing ENTRYPOINT to reference the absolute path /script.sh instead.

Cron, execute bash script as root, but one part (Python script) as user

I need to run a bash script periodically on a Jetson Nano (so, Ubuntu 18.04). The script should run system updates, pull some Python code from a repository, and run it as a specified user.
So, I created this script:
#! /bin/bash
## system updates
sudo apt update
sudo apt upgrade
## stop previous instances of the Python code
pkill python3
## move to python script folder
cd /home/user_name/projects/my_folder
## pull updates from repo
git stash
git pull
## create dummy folder to check bash script execution to this point
sudo -u user_name mkdir /home/user_name/projects/dummy_folder_00
## launch python script
sudo -u user_name /usr/bin/python3 python_script.py --arg01 --arg02
## create dummy folder to check bash script execution to this point
sudo -u user_name mkdir /home/user_name/projects/dummy_folder_01
I created a cron job running this script as root, by using
sudo crontab -e
and adding the entry
00 13 * * * /home/user_name/projects/my_folder/script.sh
Now, I can see that at the configured time, both the dummy folders are created, and they actually belong to user_name. However, the Python script isn't launched.
I tried creating the cron job as non root user (crontab -e), but at this point even if the Python script gets exectured, I guess I wouldn't be able to run apt update/upgrade.
How can I fix this?

Well, if the dummy folders did get created, that means the sudo statements work, so i'd say theres a 99%+ chance that python was infact started.
I'm guessing the problem is that you havent specified the path for the python file, and your working directory likely isn't what you're expecting it to be.
change:
sudo -u user_name /usr/bin/python3 python_script.py --arg01 --arg02
to something like
sudo -u user_name /usr/bin/python3 /path/to/your/python_script.py --arg01 --arg02
then test.
If that didn't solve the problem , then enable some logging, change the line to:
sudo -u user_name /usr/bin/python3 /path/to/your/python_script.py --arg01 --arg02 \
1> /home/user_name/projects/dummy_folder_00/log.txt 2>&1 ;
and test again, it should log STDOUT and STDERR to that file then.

crontab won't run os.system python command

Using ubuntu's 16.04 crontab and #reboot to run python3 script. The script runs properly on reboot as I see the logged output. However, my script's os.system command is not running. It runs fine if ran outside of crontab. My scripts are all executable.
crontab -l output:
SHELL=/bin/bash
#reboot nohup /usr/bin/python3 -u /home/path/scheduler.py >> /path/log.out &
scheduler.py code:
#...(check if web server is running...if not restart)
os.system('nohup /usr/bin/python3 -u /path/webserver/main.py &')
print('this function ran')
When I logged the output of the os.system command , there was no output.
As a side note, I am running python schedule commands to check the general health of a webserver. crontab doesn't seem to be the right tool for this so I just use crontab to start my python scheduler on reboot.
I am using flask as the webserver, and would use gunicorn and systemctrl if I could get it to work... but it didn't so this is my workaround.

The point is that, the command called by os.system is not in default path.
For example, tcpdump is not in /usr/bin/.
So, you can solve the problem by adding the full path of the command.

I was facing the same issue when we try to run python script directly in crontab it just by passes the os.system() commands.
Make launcher.sh:
#!bin/bash
cd /home/pi/
sudo python example.py
Then, make your script executable:
chmod 755 launcher.sh
And at last, add your script to crontab:
crontab -e
and add this line at the end:
#reboot sh /home/pi/launcher.sh
(I set the program to run at each reboot)

Docker interactive mode and executing script

I have a Python script in my docker container that needs to be executed, but I also need to have interactive access to the container once it has been created ( with /bin/bash ).
I would like to be able to create my container, have my script executed and be inside the container to see the changes/results that have occurred (no need to manually execute my python script).
The current issue I am facing is that if I use the CMD or ENTRYPOINT commands in the docker file I am unable to get back into the container once it has been created. I tried using docker start and docker attach but I'm getting the error:
sudo docker start containerID
sudo docker attach containerID
"You cannot attach to a stepped container, start it first"
Ideally, something close to this:
sudo docker run -i -t image /bin/bash python myscript.py
Assume my python script contains something like (It's irrelevant what it does, in this case it just creates a new file with text):
open('newfile.txt','w').write('Created new file with text\n')
When I create my container I want my script to execute and I would like to be able to see the content of the file. So something like:
root#66bddaa892ed# sudo docker run -i -t image /bin/bash
bash4.1# ls
newfile.txt
bash4.1# cat newfile.txt
Created new file with text
bash4.1# exit
root#66bddaa892ed#
In the example above my python script would have executed upon creation of the container to generate the new file newfile.txt. This is what I need.

My way of doing it is slightly different with some advantages.
It is actually multi-session server rather than script but could be even more usable in some scenarios:
# Just create interactive container. No start but named for future reference.
# Use your own image.
docker create -it --name new-container <image>
# Now start it.
docker start new-container
# Now attach bash session.
docker exec -it new-container bash
Main advantage is you can attach several bash sessions to single container. For example I can exec one session with bash for telling log and in another session do actual commands.
BTW when you detach last 'exec' session your container is still running so it can perform operations in background

You can run a docker image, perform a script and have an interactive session with a single command:
sudo docker run -it <image-name> bash -c "<your-script-full-path>; bash"
The second bash will keep the interactive terminal session open, irrespective of the CMD command in the Dockerfile the image has been created with, since the CMD command is overwritten by the bash - c command above.
There is also no need to appending a command like local("/bin/bash") to your Python script (or bash in case of a shell script).
Assuming that the script has not yet been transferred from the Docker host to the docker image by an ADD Dockerfile command, we can map the volumes and run the script from there:
sudo docker run -it -v <host-location-of-your-script>:/scripts <image-name> bash -c "/scripts/<your-script-name>; bash"
Example: assuming that the python script in the original question is already on the docker image, we can omit the -v option and the command is as simple as follows:
sudo docker run -it image bash -c "python myscript.py; bash"

Why not this?
docker run --name="scriptPy" -i -t image /bin/bash python myscript.py
docker cp scriptPy:/path/to/newfile.txt /path/to/host
vim /path/to/host
Or if you want it to stay on the container
docker run --name="scriptPy" -i -t image /bin/bash python myscript.py
docker start scriptPy
docker attach scriptPy
Hope it was helpful.

I think this is what you mean.
Note: THis uses Fabric (because I'm too lazy and/or don't have the time to work out how to wire up stdin/stdout/stderr to the terminal properly but you could spend the time and use straight subprocess.Popen):
Output:
$ docker run -i -t test
Entering bash...
[localhost] local: /bin/bash
root#66bddaa892ed:/usr/src/python# cat hello.txt
Hello World!root#66bddaa892ed:/usr/src/python# exit
Goodbye!
Dockerfile:
# Test Docker Image
FROM python:2
ADD myscript.py /usr/bin/myscript
RUN pip install fabric
CMD ["/usr/bin/myscript"]
myscript.py:
#!/usr/bin/env python
from __future__ import print_function
from fabric.api import local
with open("hello.txt", "w") as f:
f.write("Hello World!")
print("Entering bash...")
local("/bin/bash")
print("Goodbye!")

Sometimes, your python script may call different files in your folder, like another python scripts, CSV files, JSON files etc.
I think the best approach would be sharing the dir with your container, which would make easier to create one environment that has access to all the required files
Create one text script
sudo nano /usr/local/bin/dock-folder
Add this script as its content
#!/bin/bash
echo "IMAGE = $1"
## image name is the first param
IMAGE="$1"
## container name is created combining the image and the folder address hash
CONTAINER="${IMAGE}-$(pwd | md5sum | cut -d ' ' -f 1)"
echo "${IMAGE} ${CONTAINER}"
# remove the image from this dir, if exists
## rm remove container command
## pwd | md5 get the unique code for the current folder
## "${IMAGE}-$(pwd | md5sum)" create a unique name for the container based in the folder and image
## --force force the container be stopped and removed
if [[ "$2" == "--reset" || "$3" == "--reset" ]]; then
echo "## removing previous container ${CONTAINER}"
docker rm "${CONTAINER}" --force
fi
# create one special container for this folder based in the python image and let this folder mapped
## -it interactive mode
## pwd | md5 get the unique code for the current folder
## --name="${CONTAINER}" create one container with unique name based in the current folder and image
## -v "$(pwd)":/data create ad shared volume mapping the current folder to the /data inside your container
## -w /data define the /data as the working dir of your container
## -p 80:80 some port mapping between the container and host ( not required )
## pyt#hon name of the image used as the starting point
echo "## creating container ${CONTAINER} as ${IMAGE} image"
docker create -it --name="${CONTAINER}" -v "$(pwd)":/data -w /data -p 80:80 "${IMAGE}"
# start the container
docker start "${CONTAINER}"
# enter in the container, interactive mode, with the shared folder and running python
docker exec -it "${CONTAINER}" bash
# remove the container after exit
if [[ "$2" == "--remove" || "$3" == "--remove" ]]; then
echo "## removing container ${CONTAINER}"
docker rm "${CONTAINER}" --force
fi
Add execution permission
sudo chmod +x /usr/local/bin/dock-folder
Then you can call the script into your project folder calling:
# creates if not exists a unique container for this folder and image. Access it using ssh.
dock-folder python
# destroy if the container already exists and replace it
dock-folder python --replace
# destroy the container after closing the interactive mode
dock-folder python --remove
This call will create a new python container sharing your folder. This makes accessible all the files in the folder as CSVs or binary files.
Using this strategy, you can quickly test your project in a container and interact with the container to debug it.
One issue with this approach is about reproducibility. That is, you may install something using your shell script that is required to your application run. But, this change just happened inside of your container. So, anyone that will try to run your code will have to figure out what you have done to run it and do the same.
So, if you can run your project without installing anything special, this approach may suits you well. But, if you had to install or change some things in your container to be able to run your project, probably you need to create a Dockerfile to save these commands. That will make all the steps from loading the container, making the required changes and loading the files easy to replicate.

python - running ssh/rsync commands in a script called via crontab

I have a very strange issue that I can't seem to figure out.
When I execute a python script containing the following lines while inside a SSH terminal (putty), it works fine. But the moment I run the script via crontab or even nohup python myscript >/dev/null 2>&1& it doesn't seem to execute these commands.
subprocess.call('rsync -avr /path/to/folder/. --include "delta.*" --exclude "*" -e "ssh -o StrictHostKeyChecking=no -i /path/to/key.pem" ec2-user#'+server+':/path/to/folder/', shell=True)
local('ssh -t -o StrictHostKeyChecking=no -i /path/to/key.pem ec2-user#'+server+' "sudo /usr/bin/indexer -c /path/to/sphinx.conf --merge main delta --rotate"')
Basically all the above is doing is syncing a folder with new sphinx search engine updates to a remote server, then the second line runs a remote ssh command to force the search engine to rotate updates into production.
I do have fabric installed (hence the local command) but to avoid having to fab a second file I was hoping a single line of code could allow me to execute sudo commands on a remote server.
Can someone help me out?

I found the answer, for ssh commands in a script run in the background, you need to to have -t -t to force a pseudo terminal.
Reference:
Pseudo-terminal will not be allocated because stdin is not a terminal