Develop Reference

Python process stopped without going through TRAP - Shell Script

I have a parent shell script that calls a python script. To notify in case of python script failure, have added a TRAP in shell script. But somehow, python script is getting killed/stopped for some reason without going through the TRAP function.
Request to help with the scenarios when a script can behave in such manner
Shell Script (Parent Process):
parent.sh
set -e
on_exit(){
if [ "$?" -eq 0 ]
then
echo "Success"
else
echo "Failure"
fi
}
trap on_exit EXIT
py_script=$(python child.py)
Python Script (Child Process): child.py
def func():
isDone = "false"
while isDone == "false":
print("Waiting")
try:
## GET request which sets isDone="true" on specific value
except Exception as e:
print("Something went wrong")
sys.exit(1)
time.sleep(10)
print("Completed")
Python script never prints "Something went wrong".
Is it possible that Linux is killing the process in background if it runs for around 12 hours?
EDIT:
Investigated it further and got to know that the python process was still running in the background without performing anything. When killed it manually it threw the notification.
But the question remains, in which scenario can a process go into a state without executing further lines without being killed. I am not aware of any pending state.

Kill Bash Script from within a python exception

I have a shell script calling Python inside it.
#! /bin/bash
shopt -s extglob
echo "====test===="
~/.conda/envs/my_env/bin/python <<'EOF'
import sys
import os
try:
print("inside python")
x = 2/0
except Exception as e:
print("Exception: %s" % e)
sys.exit(2)
print("at the end of python")
EOF
echo "end of script"
If I execute this, the lines below still get printed.
"end of script"
I want to exit the shell in the exception block of the python script and let the script not reach EOF
Is there a way to create and kill a subprocess in the except block above, that will kill the entire shell script?
Can I spawn a dummy subprocess and kill it inside the exception block there by killing the entire shell script?
Any examples would be helpful.
Thanks in advance.

The whole EOF ... EOF block gets executed within the Python runtime so exiting from it doesn't affect the bash script. You'll need to collect the exit status and check it after the Python execution if you want to stop the further bash script progress, i.e.:
#!/bin/bash
~/.conda/envs/my_env/bin/python <<'EOF'
import sys
sys.exit(0x01) # use any exit code from 0-0xFF range, comment out for a clean exit
print("End of the Python script that will not execute without commenting out the above.")
EOF
exit_status=$? # store the exit status for later use
# now lets check the exit status and see if python returned a non-zero exit status
if [ $exit_status -ne 0 ]; then
echo "Python exited with a non-zero exit status, abort!"
exit $exit_status # exit the bash script with the same status
fi
# continue as usual...
echo "All is good, end of script"

From the shell script you have 2 options:
set -e: all errors quit the script
check python subcommand return code, abort if non-zero
(maybe more details here: Aborting a shell script if any command returns a non-zero value?)
Now, if you don't want to change the handling from your shell script, you could get the parent process of the python script and kill it:
except Exception as e:
import os,signal,sys
print("Exception: %s" % e)
os.kill(os.getppid(),signal.SIGTERM)
sys.exit(2)
if you need this on windows, this doesn't work (os.kill doesn't exist), you have to adapt it to invoke taskkill:
subprocess.call(["taskkill","/F","/PID",str(os.getppid())])
Now I would say that killing the parent process is bad practice. Unless you don't control the code of this parent process, you should try to handle the exit gracefully.

One way to kill the entire script could be to save the PID and then using Python's system commands to execute a kill command on the PID when the exception happens. If we imported 'os' it would be something along the lines of:
# In a shell
PID=$$
...
// Some Python Exception happens
os.system('kill -9' + $PID)

timeout limit for holding exit status from system in perl/python

I have a simple perl script that calls another python script to do the deployment of a server in cloud .
I capture the exit status of the deployment inside perl to take any further action after success/failure setup.
It's like:
$cmdret = system("python script.py ARG1 ARG2");
Here the python script runs for 3hrs to 7 hrs.
The problem here is that, irrespective of the success or failure return status, the system receive a Signal HUP at this step randomly even if the process is running in backened and breaks the steps further.
So does anyone know, if there is any time limit for holding the return status from the system which leads to sending Hangup Signal?
Inside the python script script.py, pexpect is used execute scripts remotely:
doSsh(User,Passwd,Name,'cd '+OutputDir+';python host-bringup.py setup')
doSsh(User,Passwd,Name,'cd '+OpsHome+'/ops/hlevel;python dshost.py start')
....
And doSsh is a pexpect subroutine:
def doSsh(user,password,host,command):
try:
child = pexpect.spawn("ssh -o ServerAliveInterval=100 -n %s#%s '%s'" % (user,host,command),logfile=sys.stdout,timeout=None)
i = child.expect(['password:', r'\(yes\/no\)',r'.*password for paasusr: ',r'.*[$#] ',pexpect.EOF])
if i == 0:
child.sendline(password)
elif i == 1:
child.sendline("yes")
child.expect("password:")
child.sendline(password)
data = child.read()
print data
child.close()
return True
except Exception as error:
print error
return False
This first doSsh execution takes ~6 hours and this session is killed after few hours of execution with the message : Signal HUP caught; exitingbut
the execution python host-bringup.py setup still runs in the remote host.
So in the local system, the next doSsh never runs and also the rest steps inside the perl script never continue.

SIGHUP is sent when the terminal disconnects. When you want to create a process that's not tied to the terminal, you daemonize it.
Note that nohup doesn't deamonize.
$ nohup perl -e'system "ps", "-o", "pid,ppid,sid,cmd"'
nohup: ignoring input and appending output to `nohup.out'
$ cat nohup.out
PID PPID SID CMD
21300 21299 21300 -bash
21504 21300 21300 perl -esystem "ps", "-o", "pid,ppid,sid,cmd"
21505 21504 21300 ps -o pid,ppid,sid,cmd
As you can see,
perl's PPID is that of the program that launched it.
perl's SID is that of the program that launched it.
Since the session hasn't changed, the terminal will send SIGHUP to perl when it disconnects as normal.
That said, nohup changes how perl's handles SIGHUP by causing it to be ignored.
$ perl -e'system "kill", "-HUP", "$$"; print "SIGHUP was ignored\n"'
Hangup
$ echo $?
129
$ nohup perl -e'system "kill", "-HUP", "$$"; print "SIGHUP was ignored\n"'
nohup: ignoring input and appending output to `nohup.out'
$ echo $?
0
$ tail -n 1 nohup.out
SIGHUP was ignored
If perl is killed by the signal, it's because something changed how perl handles SIGHUP.
So, either daemonize the process, or have perl ignore use SIGHUP (e.g. by using nohup). But if you use nohup, don't re-enable the default SIGHUP behaviour!

If your goal is to make your perl program ignore the HUP signal, you likely just need to set the HUP entry of the $SIG global signal handler hash:
$SIG{ 'HUP' } = 'IGNORE';
for gory details, see
perldoc perlipc

Running a bash shell from python: properly detecting when the prompt returns

I am writing a program that does the submission and bookkeeping of tasks to an external computing grid. In order to submit such a task, I need to setup the correct environment (read: execute a bash setup script) and then execute the bash command to submit the task. To complicate the matter, the task being submitted can rely on customized code which needs to be compiled locally in order to be tested, before being uploaded to grid. The compilation takes a certain amount of time, and the compiler produces output to the bash shell at unpredictable and variable intervals. You'll see how this is relevant by looking at my attempt to implement a solution:
## ---------------------------------------------------------
def shell_command(poll, shell, command):
"""
Sends a command to the shell
"""
output = ''
## Send command
shell.stdin.write(command + '\n')
shell.stdin.flush()
## Wait for output
while poll.poll(500):
result = shell.stdout.readline()
## Print and record output
print result,
output += result
return output
## ---------------------------------------------------------
def start_shell():
"""
Starts the shell associated to this job
"""
## Start the shell
shell = subprocess.Popen(['bash'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
## Associate poll to shell
poll = select.poll()
poll.register(shell.stdout.fileno(), select.POLLIN)
## Setup environment
shell_command(poll, shell, 'export ENVIRONMENT_VARIABLE=/path/to/stuff')
shell_command(poll, shell, 'source $ENVIRONMENT_VARIABLE/setup.sh')
return poll, shell
## Main code
poll, shell = start_shell()
shell_command(poll, shell, 'compile local code')
[...] do some testing on the compiled code [...]
shell_command(poll, shell, 'submit task on the grid')
So the issue I encounter is that the correct execution of the code depends on the timeout I give to poll.poll(timeout). I can always give a ridiculously long timeout, and then the code never fails, but it takes a correspondingly long time before the code finishes. With a short timeout, the execution of the code will be interrupted as soon as the compiler provides no output for longer than timeout.
I tried using subprocess.Popen.communicate(), but it doesn't seem to allow me to pass multiple commands to the same shell (and allow me to keep the shell alive for later), and I don't want to have to setup the environment every time I need to issue a new command.
It seems to me that select.poll can only detect when output is produced on stdout, but what I would really like to do is detect the prompt return. Is this possible in this context? Any other ideas?

Turns out pexpect was present in the environment in which I wish my code to run, so I tried it but with minimal success, and it would take too long to explain why. I'm satisfied for the moment with modifying the shell_command function like this:
## ---------------------------------------------------------
def shell_command(poll, shell, command):
"""
Sends a command to the shell
"""
output = ''
## Send command
shell.stdin.write(command + '; echo awesomeapplesauce\n')
shell.stdin.flush()
## Wait for end of process, signaled by awesomeapplesauce
print 'executing \'{0}\''.format(command)
while True:
if poll.poll(500):
result = shell.stdout.readline()
output += result
if 'awesomeapplesauce' in result: break
print result,
return output
It's a bit of a hack I guess, but it's sufficiently robust for my purposes. In words, at the end of every command sent to the shell, chain an echo command that echoes a unique string, and then wait for that string to be printed out to terminate the polling loop and move on. If anyone has a more fundamental way of spotting a prompt return, I would still be delighted to learn about it!

Does looking at the return code like this work?
from subprocess import call
out = open('/tmp/out.log', 'w')
err = open('/tmp/err.log', 'w')
ret = call('export FOO=foobar;echo $FOO', stdout=log, stderr=err, shell=True)
print ret
ret = call('test -e /tmp/whoosh', stdout=log, stderr=err, shell=True)
print ret

Signals are not always seen by all children in a process group

I have a problem with the way signals are propagated within a process group. Here is my situation and an explication of the problem :
I have an application, that is launched by a shell script (with a su). This shell script is itself launched by a python application using subprocess.Popen
I call os.setpgrp as a preexec_function and have verified using ps that the bash script, the su command and the final application all have the same pgid.
Now when I send signal USR1 to the bash script (the leader of the process group), sometimes the application see this signal, and sometimes not. I can't figure out why I have this random behavior (The signal is seen by the app about 50% of the time)
Here is he example code I am testing against :
Python launcher :
#!/usr/bin/env python
p = subprocess.Popen( ["path/to/bash/script"], stdout=…, stderr=…, preexec_fn=os.setpgrp )
# loop to write stdout and stderr of the subprocesses to a file
# not that I use fcntl.fcntl(p.stdXXX.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
p.wait()
Bash script :
#!/bin/bash
set -e
set -u
cd /usr/local/share/gios/exchange-manager
CONF=/etc/exchange-manager.conf
[ -f $CONF ] && . $CONF
su exchange-manager -p -c "ruby /path/to/ruby/app"
Ruby application :
#!/usr/bin/env ruby
Signal.trap("USR1") do
puts "Received SIGUSR1"
exit
end
while true do
sleep 1
end
So I try to send the signal to the bash wrapper (from a terminal or from the python application), sometimes the ruby application will see the signal and sometimes not. I don't think it's a logging issue as I have tried to replace the puts by a method that write directly to a different file.
Do you guys have any idea what could be the root cause of my problem and how to fix it ?

Your signal handler is doing too much. If you exit from within the signal handler, you are not sure that your buffers are properly flushed, in other words you may not be exiting gracefully your program. Be careful of new signals being received when the program is already inside a signal handler.
Try to modify your Ruby source to exit the program from the main loop as soon as an "exit" flag is set, and don't exit from the signal handler itself.
Your Ruby application becomes:
#!/usr/bin/env ruby
$done = false
Signal.trap("USR1") do
$done = true
end
until $done do
sleep 1
end
puts "** graceful exit"
Which should be much safer.
For real programs, you may consider using a Mutex to protect your flag variable.