Develop Reference

Reload or restart python script in a bash shell

I'm a brand new noob in python universe, so don't judge me too fast :-)
I'm trying to force a python script to reload or restart at the beggining of a bash script.
I've tried :
pkill -f myscript.py
and
killall myscript.py
and others...
Actually, I would like to make run the same script that call .wav files after having changed those .wav files... If I don't reload the script or restart it, it keeps playing the old files.
Maybe, there is other solutions.
Here is the script I want to reload (it's a button script playing music for my daughter)
#!/usr/bin/env python3
import pygame
from gpiozero import LED, Button
from signal import pause
pygame.init()
button_sounds = {Button(2): pygame.mixer.Sound("/home/pi/gpio-music-box/samples/1.wav"),
Button(3): pygame.mixer.Sound("/home/pi/gpio-music-box/samples/2.wav"),
Button(4): pygame.mixer.Sound("/home/pi/gpio-music-box/samples/3.wav"),
Button(17): pygame.mixer.Sound("/home/pi/gpio-music-box/samples/4.wav")}
for button, sound in button_sounds.items():
button.when_pressed = sound.play
pause()
And here is my bash script :
#!/bin/bash
***HERE THE COMMAND I NEED !***
rm -r /home/pi/gpio-music-box/samples/*
cp -r /home/pi/gpio-music-box/comptines/* /home/pi/gpio-music-box/samples/
/home/pi/gpio-music-box/music.py
Thank you very much, and scuze my english, I'm french :-)
Andy

try this
#!/bin/bash
pid=$(ps auxwww | grep nameOfScript.py | grep -v grep | awk '{print $2}')
kill -9 $pid
rm -r /home/pi/gpio-music-box/samples/*
cp -r /home/pi/gpio-music-box/comptines/* /home/pi/gpio-music-box/samples/
nohup /home/pi/gpio-music-box/music.py &
Have a nice day

Firstly, you can reduce a lot of the "noise" from ps by using output formatting. You can then stop the need for using both grep and awk by using awk to do the searching also.
ps -eo "%p %a" | awk '/nameOfScript.py/ && $1 != PROCINFO["pid"] { print "kill -9 "$1 }'
This forces ps to only print the pid (%p) and the full command (%a). The output is then piped to awk where is searches for lines with the name of the script contained. It discounts any entries with the current process id of awk and then uses this to print a kill command with the relevant process id.
Once you have verified that the kill command displays as expected, you can use awk's system function to actually run the command through:
ps -eo "%p %a" | awk '/prometheous-things.py/ && $1 != PROCINFO["pid"] { system("kill -9 "$1) }'

Running python scripts for different input directory through bash terminal

I am trying to automate my task through the terminal using bash. I have a python script that takes two parameters (paths of input and output) and then the script runs and saves the output in a text file.
All the input directories have a pattern that starts from "g-" whereas the output directory remains static.
So, I want to write a script that could run on its own so that I don't have to manually run it on hundreds of directories.
$ python3 program.py ../g-changing-directory/ ~/static-directory/ > ~/static-directory/final/results.txt

You can do it like this:
find .. -maxdepth 1 -type d -name "g-*" | xargs -n1 -P1 -I{} python3 program.py {} ~/static-directory/ >> ~/static-directory/final/results.txt
find .. will look in the parent directory -maxdepth 1 will look only on the top level and not take any subdirectories -type d only takes directories -name "g-*" takes objects starting with g- (use -iname "g-*" if you want objects starting with g- or G-).
We pipe it to xargs which will apply the input from stdin to the command specified. -n1 tells it to start a process per input word, -P1 tells it to only run one process at a time, -I{} tells it to replace {} with the input in the command.
Then we specify the command to run for the input, where {} is replaced by xargs.: python3 program.py {} ~/static-directory/ >> ~/static-directory/final/results.txt have a look at the >> this will append to a file if it exists, while > will overwrite the file, if it exists.
With -P4 you could start four processes in parallel. But you do not want to do that, as you are writing into one file and multi-processing can mess up your output file. If every process would write into its own file, you could do multi-processing safely.
Refer to man find and man xargs for further details.
There are many other ways to do this, as well. E.g. for loops like this:
for F in $(ls .. | grep -oP "g-.*"); do
python3 program.py $F ~/static-directory/ >> ~/static-directory/final/results.txt
done

There are many ways to do this, here's what I would write:
find .. -type d -name "g-*" -exec python3 program.py {} ~/static-directory/ \; > ~/static-directory/final/results.txt
You haven't mentioned if you want nested directories to be included, if the answer is no then you have to add the -maxdepth parameter as in #toydarian's answer.

Evaluate "Shell command line with shell variables" from python OR evaluate python string as shell command line

CONTEXT
I am working on a simulation cluster.
In order to make as flexible as possible (working with different simulation soft) , we created a python file that parse a config file defining environment variables, and command line to start the simulation. This command is launched through SLURM sbatch command (shell $COMMAND)
ISSUE
From python, all Environment variables are enrolled reading the config file
I have issue with variable COMMAND that is using other environment variables (displayed as shell variable)
For example
COMMAND = "fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE"
os.environ['COMMAND']=COMMAND
NUMPROCS = "32"
os.environ['NUMPROCS']=NUMPROCS
[...]
exe = Popen(['sbatch','template_document.sbatch'], stdout=PIPE, stderr=PIPE)
sbatch distribute COMMAND to all simulation nodes as COMMAND being a command line
COMMAND recalls other saved env. variables. Shell interprets it strictly as text... Which makes the command line fails. it is strictly as a string using $ not variable for example :
'fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE'
SOLUTION I AM LOOKING FOR
I am looking for a simple solution
Solution 1: A 1 to 3 python command lines to evaluate the COMMAND as shell command to echo
Solution 2: A Shell command to evaluate the variables within the "string" $COMMAND as a variable
At the end the command launched from within sbatch should be
fluent -3ddp -n32 -hosts=./hosts -file /path/to/JOBFILE

You have a few options:
Partial or no support for bash's variable substitution, e.g. implement some python functionality to reproduces bash's $VARIABLE syntax.
Reproduce all of bash's variable substitution facilities which are supported in the config file ($VARIABLE, ${VARIABLE}, ${VARIABLE/x/y}, $(cmd) - whatever.
Let bash do the heavy lifting, for the cost of performance and possibly security, depending on your trust of the content of the config files.
I'll show the third one here, since it's the most resilient (again, security issues notwithstanding). Let's say you have this config file, config.py:
REGULAR = "some-text"
EQUALS = "hello = goodbye" # trap #1: search of '='
SUBST = "decorated $REGULAR"
FANCY = "xoxo${REGULAR}xoxo"
CMDOUT = "$(date)"
BASH_A = "trap" # trap #2: avoid matching variables like BASH_ARGV
QUOTES = "'\"" # trap #3: quoting
Then your python program can run the following incantation:
bash -c 'source <(sed "s/^/export /" config.py | sed "s/[[:space:]]*=[[:space:]]*/=/") && env | grep -f <(cut -d= -f1 config.py | grep -E -o "\w+" | sed "s/.*/^&=/")'
which will produce the following output:
SUBST=decorated some-text
CMDOUT=Thu Nov 28 12:18:50 PST 2019
REGULAR=some-text
QUOTES='"
FANCY=xoxosome-textxoxo
EQUALS=hello = goodbye
BASH_A=trap
Which you can then read with python, but note that the quotes are now gone, so you'll have to account for that.
Explanation of the incantation:
bash -c 'source ...instructions... && env | grep ...expressions...' tells bash to read & interpret the instructions, then grep the environment for the expressions. We're going to turn the config file into instructions which modify bash's environment.
If you try using set instead of env, the output will be inconsistent with respect to quoting. Using env avoids trap #3.
Instructions: We're going to create instructions for the form:
export FANCY="xoxo${REGULAR}xoxo"
so that bash can interpret them and env can read them.
sed "s/^/export /" config.py prefixes the variables with export.
sed "s/[[:space:]]*=[[:space:]]*/=/" converts the assignment format to syntax that bash can read with source. Using s/x/y/ instead of s/x/y/g avoids trap #1.
source <(...command...) causes bash to treat the output of the command as a file and run its lines, one by one.
Of course, one way to avoid this complexity is to have the file use bash syntax to begin with. If that were the case, we would use source config.sh instead of source <(...command...).
Expressions: We want to grep the output of env for patterns like ^FANCY=.
cut -d= -f1 config.py | grep -E -o "\w+" finds the variable names in config.py.
sed "s/.*/^&=/" turns variable names like FANCY to grep search expressions such as ^FANCY=. This is to avoid trap #2.
grep -f <(...command...) gets grep to treat the output of the command as a file containing one search expression in each line, which in this case would be ^FANCY=, ^CMDOUT= etc.
EDIT
Since you actually want to just pass this environment to another bash command rather than use it in python, you can actually just have python run this:
bash -c 'source <(sed "s/^/export /" config.py | sed "s/[[:space:]]*=[[:space:]]*/=/") && $COMMAND'
(assuming that COMMAND is specified in the config file).

It seems I have not explained well enough the issue, but your 3rd solution seems replying to my expectations... though so far I did not manage to adapt it
Based on your 3rd solution BASH, I will make it more straight :
Let's say I have got following after running python, and this that cannot be modified
EXPORT COMMAND='fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE'
EXPORT JOBFILE='/path/to/jobfile'
EXPORT NUMPROCS='32'
EXPORT WHATSOEVER='SPECIFIC VARIABLE TO SIMULATION SOFTWARE'
I wish to execute the following from the slurm batch file (bash), using $COMMAND / $JOBFILE /$NUMPROCS
fluent -3ddp -n32-hosts=./hosts -file /path/to/jobfile
Please note : I have backup solution in python - I managed to substitute $VARIABLE by its value - based on the assumption $VARIABLE is not composed by another $variable... using regex substitution... just it looks so many lines to have what seemed to me simple request

Restart python script automatically even when it crashes in Linux

I have a python program that has to be running all the time. If for some reason it was stopped I want to restart it automatically. I thought of having a cron that will run every n number of seconds and check the program is running. My shell script is looks like this:
#!/usr/bin/env bash
CM_COMMAND=`ps aux| grep abc| grep def| grep sudo`
LEN_COMMAND=${#CM_COMMAND}
if[["$LEN_COMMAND" -le "5"]]
then
echo "start the python program"
fi
exit
When I run this script I am getting the error: my_prog.sh: line 4: $'if[[118\r -le 5]]\r': command not found'
What is the alternative of doing this and what is the problem with my script?

Maybe this would be more robust?
1) save the PID of your process when you start it with:
{your_python_command} & echo $! >>/{some_folder}/your_app.pid
2) This script will check and restart if it can't find the PID..
#!/usr/bin/env bash
PID=`cat /{some_folder}/your_app.pid`
if ! ps -p $PID > /dev/null
then
rm /{some_folder}/your_app.pid
{your_python_command} & echo $! >>/{some_folder}/your_app.pid
fi
3) To add it to a cronjob:
crontab -e
choose your text editor and add this row at the end of the file:
*/1 * * * * /{your_path}/{your_script_name}
exit and save
(this will run the script every minute, check crontab manual to set your exact interval)

How about making it a service? A very clean solution, in my opinion.
For more information on how to do it, you can read this article.

Restart kivy-program respectively python with inotify

I fear that my question is a duplicate but I can't find the answer. Maybe you can help me?
I would like to restart my kivy-program if I save the kv or py file.
I tried with
inotifywait -mq -e close_write /home/name/kivy/ | while read FILE
do
pkill python
python /home/name/kivy/main.py
done
If I change a file the first time, main.py starts, but if I change it again I need to close the program by hand before it restarts.
Instead of pkill python I also tried to use
kill $(ps aux | pgrep '[p]ython' | awk '{print $2}')
but with the same result and the problem that the mintMenu.py is closing, too.
Should I use something totally different to inotify?

I'm using entr to achieve the same thing. Once installed (e.g. via brew), just run the following command in your work directory /home/name/kivy/:
find . -name "*.py" -or -name "*.kv" | entr sh -c "pkill -f python main.py ; python main.py &"