I have a python script which I want to run as a daemon under daemontools. Usually the service running under deamontools runs in foreground and if it goes down, daemontools will restart it. The service should also handle signals as described here.
My program catches SIGINT and SIGALRM and when the signals are received program exits.
When the program is run from command line and kill -1 <pid> and kill -15 <pid> is signaled, the signal handler is run, which causes program to exit, and corresponding logs are printed.
But when the program is run under daemontools, and svc -d /service/myrogram is executed, the program neither exits nor logs are printed.
I am using following run script to run the program under daemontools
#!/bin/sh
exec 2>&1
/usr/bin/python /home/test/sigtest.py
I am wondering why kill -15 <pid> works while svc -d /service/myrogram does not seems to deliver the signal to the python program.
The python script I am working with is:
from pyudev import Context, Monitor, MonitorObserver
import signal, sys, time
def print_device_event(device):
print ('New event - {0.action}: {0.device_path}'.format(device))
if __name__ == '__main__':
context = Context()
monitor = Monitor.from_netlink(context)
observer = MonitorObserver(monitor, callback=print_device_event, name='monitor-observer')
print ("Processing started")
observer.start()
def handler(signum, frame):
print("Received Signal: %d"%signum)
observer.send_stop()
print("Exiting.")
signal.signal(signal.SIGTERM, handler)
signal.signal(signal.SIGHUP, handler)
try:
while observer.is_alive():
observer.join(timeout=1.0)
except (KeyboardInterrupt, SystemExit):
print("Caught KeyboardInterrupt, exiting")
observer.send_stop()
I am using following run script to run the program under daemontools
#!/bin/sh
exec 2>&1
/usr/bin/python /home/test/sigtest.py
And there's your error. You're forking a child process to run the python program in. Stop your daemon from forking. Daemons do not need to, and should not, fork to run in a child process, be they running under daemontools, daemontools-encore, runit, s6, perp, or even systemd.
The standard operation of the shell, remember, is to fork and run commands in child processes.
For best results, you should acclimate yourself to writing run scripts using something other than the shell, that chain loads daemon programs. Here's your run script using Laurent Bercot's execline instead of /bin/sh as the script interpreter:
#!/command/execlineb -PW
fdmove -c 2 1
/usr/bin/python /home/test/sigtest.py
execline doesn't have any of the overhead of all of the things that even non-interactive shells do, including parsing startup "rc" files. And it comes with a useful suite of chain loading utilities for doing many of the things that one might want to do in run scripts.
But even execline does things that you don't need in a run script. Hence the -P option to turn off argument pushing, which has no utility for run scripts. Here's your run script using an even more barebones script interpreter, my nosh program:
#!/usr/local/bin/nosh
fdmove -c 2 1
/usr/bin/python /home/test/sigtest.py
Finally, here's your run script for /bin/sh, corrected:
#!/bin/sh
exec 2>&1
exec /usr/bin/python /home/test/sigtest.py
Further reading
https://superuser.com/a/708384/38062
Russ Allbery (2013-12-18). Bug#727708: upstart proposed policy in Debian. debian-ctte.
Jonathan de Boyne Pollard (2001). "Don't fork() in order to 'put the dæmon into the background'.". Mistakes to avoid when designing Unix dæmon programs. Frequently Given Answers.
Related
I am writing a python script that calls a bash script.
from subprocess import call
rc = call("./try_me.sh")
How can I exit the running bash file without exiting the running python script?
I need something like Ctrl + C.
There's different ways to approach this problem.
It appears that you're writing some sort of CLI tool since you referenced Ctrl+C. If that's the case you can use & and send a SIGINT signal to stop it when you need.
import os
os.system('nohup ./try_me.sh &')
If you want stricter control try using async subprocess management:
https://docs.python.org/3/library/asyncio-subprocess.html
After Some research, I found I should have used Popen to run the bash file as #AKX suggested.
from subprocess import Popen
r1 = Popen('printf "*Systempassword* \n" |sudo -S ./try_me.sh &', shell=True, preexec_fn=os.setsid))
when you need to stop running the bash file
import os
os.killpg(os.getpgid(r1.pid), signal.SIGTERM) # Send the signal to all the process groups
I have a parent script (start.py) who's primary purpose is to start background processes and exit. When I ssh directly to remote_host and run the script, it works as expected.
[user#local_host ~]# ssh remote_host
user#remote_host's password: ****
[user#remote_host ~]# time python start.py --config_file /data/workload.pg
real 0m0.037s
user 0m0.025s
sys 0m0.012s
The exit code of this script:
[root#perf72 ~]# echo $?
0
To simplify, instead of establishing the ssh session first and running the command, I want to just execute the command remotely from local_host:
[user#local_host ~]# ssh -o StrictHostKeyChecking=no -i /tmp/tmpqcz5l5il user#remote_host -p 22 "python start.py --config_file /data/workload.pg"
real 12m6.594s
user 0m0.027s
sys 0m0.016s
The problem here is that the ssh session remains open during the life of the background processes and not the life of the start.py script which is less than one second. It should just disconnect when the start.py script exits, but it doesn't.
Do I need a specific sys.exit() signal in the start.py script which will prompt the ssh session to disconnect?
ssh is awaiting output on the called process's stdout, so it can print it if there is any. That file handle is inherited by the subprocesses you're spawning, so it's still open even though the python script has exited, and as long as it's open, ssh will keep waiting.
If you change your ssh command line to run the remote script as "python start.py --config_file /data/workload.pg > /dev/null" instead, the ssh connection will close as soon as the python script does.
I want to make sure my python script is always running, 24/7. It's on a Linux server. If the script crashes I'll restart it via cron.
Is there any way to check whether or not it's running?
Taken from this answer:
A bash script which starts your python script and restarts it if it didn't exit normally :
#!/bin/bash
until script.py; do
echo "'script.py' exited with code $?. Restarting..." >&2
sleep 1
done
Then just start the monitor script in background:
nohup script_monitor.sh &
Edit for multiple scripts:
Monitor script:
cat script_monitor.sh
#!/bin/bash
until ./script1.py
do
echo "'script1.py' exited with code $?. Restarting..." >&2
sleep 1
done &
until ./script2.py
do
echo "'script2.py' exited with code $?. Restarting..." >&2
sleep 1
done &
scripts example:
cat script1.py
#!/usr/bin/python
import time
while True:
print 'script1 running'
time.sleep(3)
cat script2.py
#!/usr/bin/python
import time
while True:
print 'script2 running'
time.sleep(3)
Then start the monitor script:
./script_monitor.sh
This starts one monitor script per python script in the background.
Try this and enter your script name.
ps aux | grep SCRIPT_NAME
Create a script (say check_process.sh) which will
Find the process id for your python script by using ps command.
Save it in a variable say pid
Create an infinite loop. Inside it, search for your process. If found then sleep for 30 or 60 seconds and check again.
If pid not found, then exit the loop and send mail to your mail_id saying that process is not running.
Now call check_process.sh by a nohup so it will run in background continuously.
I implemented it way back and remember it worked fine.
You can use
runit
supervisor
monit
systemd (i think)
Do not hack this with a script
upstart, on Ubuntu, will monitor your process and restart it if it crashes. I believe systemd will do that too. No need to reinvent this.
I can't figure out how to close a bash shell that was started via Popen. I'm on windows, and trying to automate some ssh stuff. This is much easier to do via the bash shell that comes with git, and so I'm invoking it via Popen in the following manner:
p = Popen('"my/windows/path/to/bash.exe" | git clone or other commands')
p.wait()
The problem is that after bash runs the commands I pipe into it, it doesn't close. It stays open causing my wait to block indefinitely.
I've tried stringing an "exit" command at the end, but it doesn't work.
p = Popen('"my/windows/path/to/bash.exe" | git clone or other commands && exit')
p.wait()
But still, infinite blocking on the wait. After it finishes its task, it just sits at a bash prompt doing nothing. How do I force it to close?
Try Popen.terminate() this might help kill your process. If you have only synchronous executing commands try to use it directly with subprocess.call().
for example
import subprocess
subprocess.call(["c:\\program files (x86)\\git\\bin\\git.exe",
"clone",
"repository",
"c:\\repository"])
0
Following is an example of using a pipe but this is a little overcomplicated for most use cases and makes sense only if you talk with a service that needs interaction (at least in my opinion).
p = subprocess.Popen(["c:\\program files (x86)\\git\\bin\\git.exe",
"clone",
"repository",
"c:\\repository"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
print p.stderr.read()
fatal: destination path 'c:\repository' already exists and is not an empty directory.
print p.wait(
128
This can be applied to ssh as well
To kill the process tree, you could use taskkill command on Windows:
Popen("TASKKILL /F /PID {pid} /T".format(pid=p.pid))
As #Charles Duffy said, your bash usage is incorrect.
To run a command using bash, use -c parameter:
p = Popen([r'c:\path\to\bash.exe', '-c', 'git clone repo'])
In simple cases, you could use subprocess.check_call instead of Popen().wait():
import subprocess
subprocess.check_call([r'c:\path\to\bash.exe', '-c', 'git clone repo'])
The latter command raises an exception if bash process returns non-zero status (it indicates an error).
I've written this watchdog script to monitor VLC player and kill it when playback has stopped because VLC continues to inhibit the power management daemon after playback. The script works. I can run it from the command line or through IDLE and it kills VLC when playback stops. I've added many variations of the command to start the script to my startup applications as described here but when I reboot, if it is running at all, it stops as soon as VLC starts. Restarting it from a terminal cause it to stay running and do what it is supposed to do. I don't know if this is a problem with the script or something peculiar about Ubuntu Startup Applications (although I'm leaning towards Ubuntu). Maybe something to do with permissions? (Although I did chmod +x) Should I be executing some other commands to make sure DBus is up before I launch the script? Part of me thinks that something isn't fully loaded when the script starts so I tried sleeping before launching using the *nix sleep command, the X-GNOME-Autostart-Delay, and time.sleep(n) in the python code. The pythonic way seems to have the best chance of success. The *nix ways seem to only make startup take longer and at the end of it I find that the process isn't even running. I'm using the python-setproctitle module to name the process so I can quickly see if it is running with a ps -e from terminal. I'm out of ideas and about ready to just manually run the script whenever I reboot (although in principle I think that the machine should do it for me because I told it to). Some variations of Startup Application command lines that I've tried are:
/path/to/script/vlc_watchdog.py
"/path/to/script/vlc_watchdog.py"
/path/to/script/vlc_watchdog.py &
"/path/to/script/vlc_watchdog.py &"
python /path/to/script/vlc_watchdog.py
python /path/to/script/vlc_watchdog.py &
"python /path/to/script/vlc_watchdog.py"
"python /path/to/script/vlc_watchdog.py &"
bash -c "/path/to/script/vlc_watchdog.py"
sleep 30 ; /path/to/script/vlc_watchdog.py
sleep 30 && /path/to/script/vlc_watchdog.py
etc...
Full script:
#!/usr/bin/env python
import time
time.sleep(30)
import dbus
import os
import subprocess
from subprocess import Popen, PIPE
import daemon
import setproctitle
setproctitle.setproctitle('VLC-Watchdog')
sleeptime = 5
def vlc_killer():
bus = dbus.SessionBus()
vlc_media_player_obj = bus.get_object("org.mpris.MediaPlayer2.vlc", "/org/mpris/MediaPlayer2")
props_iface = dbus.Interface(vlc_media_player_obj, 'org.freedesktop.DBus.Properties')
pb_stat = props_iface.Get('org.mpris.MediaPlayer2.Player', 'PlaybackStatus')
if pb_stat == 'Stopped':
os.system("kill -9 $(pidof vlc)")
else:
time.sleep(sleeptime)
def vlc_is_running():
ps = subprocess.Popen(['ps', '-e'], stdout = PIPE)
out, err = ps.communicate()
for line in out.splitlines():
if 'vlc' in line:
return True
return False
def run():
while True:
if vlc_is_running():
vlc_killer()
else:
time.sleep(sleeptime)
with daemon.DaemonContext():
run()
In the shell script that starts your Python code (the one in the Ubuntu startup/initialization process), use something like:
#!/bin/sh
set -x
exec > /tmp/errors.out 2>&1
/path/to/script/vlc_watchdog.py
Then after things go awry again (that is, after another reboot), inspect /tmp/errors.out to see the error messages related to whatever happened. There should be a Python traceback in there, or at least a shell error.