Develop Reference

Stop a bash script in python [duplicate]

I am currently trying to write (Python 2.7.3) kind of a wrapper for GDB, which will allow me to dynamically switch from scripted input to interactive communication with GDB.
So far I use
self.process = subprocess.Popen(["gdb vuln"], stdin = subprocess.PIPE, shell = True)
to start gdb within my script. (vuln is the binary I want to examine)
Since a key feature of gdb is to pause the execution of the attached process and allow the user to inspect registers and memory on receiving SIGINT (STRG+C) I do need some way to pass a SIGINT signal to it.
Neither
self.process.send_signal(signal.SIGINT)
nor
os.kill(self.process.pid, signal.SIGINT)
or
os.killpg(self.process.pid, signal.SIGINT)
work for me.
When I use one of these functions there is no response. I suppose this problem arises from the use of shell=True. However, at this point I am really out of ideas.
Even my old friend Google couldn't really help me out this time, so maybe you can help me. Thank's in advance.
Cheers, Mike

Here is what worked for me:
import signal
import subprocess
try:
p = subprocess.Popen(...)
p.wait()
except KeyboardInterrupt:
p.send_signal(signal.SIGINT)
p.wait()

I looked deeper into the problem and found some interesting things. Maybe these findings will help someone in the future.
When calling gdb vuln using suprocess.Popen() it does in fact create three processes, where the pid returned is the one of sh (5180).
ps -a
5180 pts/0 00:00:00 sh
5181 pts/0 00:00:00 gdb
5183 pts/0 00:00:00 vuln
Consequently sending a SIGINT to the process will in fact send SIGINT to sh.
Besides, I continued looking for an answer and stumbled upon this post
https://bugzilla.kernel.org/show_bug.cgi?id=9039
To keep it short, what is mentioned there is the following:
When pressing STRG+C while using gdb regularly SIGINT is in fact sent to the examined program (in this case vuln), then ptrace will intercept it and pass it to gdb.
What this means is, that if I use self.process.send_signal(signal.SIGINT) it will in fact never reach gdb this way.
Temporary Workaround:
I managed to work around this problem by simply calling subprocess.popen() as follows:
subprocess.Popen("killall -s INT " + self.binary, shell = True)
This is nothing more than a first workaround. When multiple applications with the same name are running might do some serious damage. Besides, it somehow fails, if shell=True is not set.
If someone has a better fix (e.g. how to get the pid of the process startet by gdb), please let me know.
Cheers, Mike
EDIT:
Thanks to Mark for pointing out to look at the ppid of the process.
I managed to narrow down the process's to which SIGINT is sent using the following approach:
out = subprocess.check_output(['ps', '-Aefj'])
for line in out.splitlines():
if self.binary in line:
l = line.split(" ")
while "" in l:
l.remove("")
# Get sid and pgid of child process (/bin/sh)
sid = os.getsid(self.process.pid)
pgid = os.getpgid(self.process.pid)
#only true for target process
if l[4] == str(sid) and l[3] != str(pgid):
os.kill(pid, signal.SIGINT)

I have done something like the following in the past and if I recollect it seemed to work for me :
def detach_procesGroup():
os.setpgrp()
subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
preexec_fn=detach_processGroup)

python - prevent IOError: [Errno 5] Input/output error when running without stdout

I have a script that runs automatically on server through cronjob and it import and run several other scripts.
Some of them use prints, which naturally creates IOError: [Errno 5] Input/output error because the script runs without any SSH / terminal connected, so there's no proper stdout setup.
There are lots of questions about this subject but I couldn't find anyone that actually solve it, assuming I can't remove the print or change the executed scripts.
I tried several things, including:
class StdOut(object):
def __init__(self):
pass
def write(self, string):
pass
sys.stdout = StdOut()
sys.stderr = StdOut()
and
from __future__ import print_function
import __builtin__
def print(*args, **kwargs):
pass
__builtin__.print = print
But none of it works. I assume it only affect the module itself and not the modules I import / run later.
So how can I create a stub stdout that will affect all modules in the process? Like I said, I don't want to change the scripts that are executed from the main module, but I can change everything inside the importing module. And just to clearify - everything is imported, no new processes are spawned etc.
Thanks,

Modifying the builtin or changing sys.stdout should work (except for subprocesses—but you ruled those out) as long as you do it early enough. If not, though, there's a lower level trick that's much easier:
run your python scripts with I/O redirection that discards output:
python foo.py >/dev/null 2>&1
(assuming Unix-y scripts, as implied by "cron" in the question)
or, redirect file descriptors 1 and 2 (same idea as above, but done within your Python startup rather than as part of the cron-invoked command):
import os
fd = os.open(os.devnull, os.O_RDWR)
# NB: even if stdin is closed, fd >= 0
os.dup2(fd, 1)
os.dup2(fd, 2)
if fd > 2:
os.close(fd)
(this particular bit of code has the side effect of making /dev/null act as stdin, if all descriptors were closed). [Edit: I started with with open(...) and then switched to os.open and did not test the final version. Fixed now.]
All that said, a good cron really should have stdout and stderr connected somewhere, and should email the output/error-output to you. Not all cron versions are this nice though.

Control a subprocess (specifically gdb) in multiple ways

I am developing a wrapper around gdb using python. Basically, I just want to be able to detect a few setup annoyances up-front and be able to run a single command to invoke gdb, rather than a huge string I have to remember each time.
That said, there are two cases that I am using. The first, which works fine, is invoking gdb by creating a new process and attaching to it. Here's the code that I have for this one:
def spawnNewProcessInGDB():
global gObjDir, gGDBProcess;
from subprocess import Popen
from os.path import join
import subprocess
binLoc = join(gObjDir, 'dist');
binLoc = join(binLoc, 'bin');
binLoc = join(binLoc, 'mycommand')
profileDir = join(gObjDir, '..')
profileDir = join(profileDir, 'trash-profile')
try:
gGDBProcess = Popen(['gdb', '--args', binLoc, '-profile', profileDir], cwd=gObjDir)
gGDBProcess.wait()
except KeyboardInterrupt:
# Send a termination signal to the GDB process, if it's running
promptAndTerminate(gGDBProcess)
Now, if the user presses CTRL-C while this is running, it breaks (i.e. it forwards the CTRL-C to GDB). This is the behavior I want.
The second case is a bit more complicated. It might be the case that I already had this program running on my system and it crashed, but was caught. In this case, I want to be able to connect to it using gdb to get a stack trace (or perhaps I was already running it, and I simply now want to connect to the process that's already in memory).
As a convenience feature, I've created a mirror function, which will connect to a running process using gdb:
def connectToProcess(procNum):
global gObjDir, gGDBProcess
from subprocess import Popen
import subprocess
import signal
print("Connecting to mycommand process number " + str(procNum) + "...")
try:
gGDBProcess = Popen(['gdb', '-p', procNum], cwd=gObjDir)
gGDBProcess.wait()
except KeyboardInterrupt:
promptAndTerminate(gGDBProcess)
Again, this seems to work as expected. It starts gdb, I can set breakpoints, run the program, etc. The only catch is that it doesn't forward CTRL-C to gdb if I press it while the program is running. Instead, it jumps immediately to promptAndTerminate().
I'm wondering if anyone can see why this is happening - the two calls to subprocess.Popen() seem identical to me, albeit that one is running gdb in a different mode.
I have also tried replacing the call to subprocess.Popen() with the following:
gGDBProcess = Popen(['gdb', '-p', procNum], cwd=gObjDir, stdin=subprocess.PIPE)
but this leads to undesirable results as well, because it doesn't actually communicate anything to the child gdb process (e.g. if I type in c to start the program running again after it is broken upon connection from gdb, it doesn't do anything). Again, it terminates the running python process when I type CTRL-C.
Any help would be appreciated!

Unable to open a Python subprocess in Web2py (SIGABRT)

I've got an Apache2/web2py server running using the wsgi handler functionality. Within one of the controllers, I am trying to run an external executable to perform some processing on 2 files.
My approach to this is to use the subprocess module to kick off the executable. I have simplified the code to a bare-bones implementation with little success.
from subprocess import *
p = Popen(("echo", "Hello"), shell=False)
ret = p.wait()
print "Process ended with status %s" % ret
When running the above code on its own (create new file and running via python command line), it works exactly as expected.
However, as soon as I place the exact same code into my web2py controller, the external process stops working. Instead of the process returning with code 0 as is expected in the above example, it always returns -6 and "Hello" is not printed to stdout.
After doing some digging, I found that negative results from p.wait() implies that a signal caused the process to end abnormally. And, according to some docs I found, -6 corresponds to the SIGABRT signal.
I would have expected this signal to be a result of some poorly executed code in my child process. However, since this is only running echo (and since it works outside of web2py) I have my doubts that the child process is signalling itself.
Is there some web2py limitation/configuration that causes Popen() requests to always fail? If so, how can I modify my logic so that the controller (or whatever) is actually able to spawn this external process?
** EDIT: Looks like web2py applications may not like the subprocess module. According to a reply to a message reply in the web2py email group:
"You should not use subprocess in a web2py application (if you really need too, look into the admin/controllers/shell.py) but you can use it in a web2py program running from shell (web2py.py -R myprogram.py)."
I will be checking out some options based on the note here and see if any solution presents itself.

In the end, the best I was able to come up with involved setting up a simple XML RPC server and call the functions from that:
my_server.py
#my_server.py
from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
from subprocess import *
proc_srvr = xmlrpclib.ServerProxy("http://localhost:12345")
def echo_fn():
p = Popen(("echo", "hello"), shell=False)
ret = p.wait()
print "Process ended with status %s" % ret
return True # RPC Server doesn't like to return None
def main():
server = SimpleXMLRPCServer(("localhost", 12345), ErrorHandler)
server.register_function(echo_fn, "echo_fn")
while True:
server.handle_request()
if __name__ == "__main__":
main()
web2py_controller.py
#web2py_controller.py
def run_echo():
proc_srvr = xmlrpclib.ServerProxy("http://localhost:12345")
proc_srvr.echo_fn()
I'll be honest, I'm not a Python nor SimpleRPCServer guru, so the overall code may not be up to best-practice standards. However, going this route did allow me to, in effect, call a subprocess from a controller in web2py.
(Note, this was a quick and dirty simplification of the code that I have in my project. I have not validated it is in a working state, so it may require some tweaks.)

Ensure a single instance of an application in Linux

I'm working on a GUI application in WxPython, and I am not sure how I can ensure that only one copy of my application is running at any given time on the machine. Due to the nature of the application, running more than once doesn't make any sense, and will fail quickly. Under Win32, I can simply make a named mutex and check that at startup. Unfortunately, I don't know of any facilities in Linux that can do this.
I'm looking for something that will automatically be released should the application crash unexpectedly. I don't want to have to burden my users with having to manually delete lock files because I crashed.

The Right Thing is advisory locking using flock(LOCK_EX); in Python, this is found in the fcntl module.
Unlike pidfiles, these locks are always automatically released when your process dies for any reason, have no race conditions exist relating to file deletion (as the file doesn't need to be deleted to release the lock), and there's no chance of a different process inheriting the PID and thus appearing to validate a stale lock.
If you want unclean shutdown detection, you can write a marker (such as your PID, for traditionalists) into the file after grabbing the lock, and then truncate the file to 0-byte status before a clean shutdown (while the lock is being held); thus, if the lock is not held and the file is non-empty, an unclean shutdown is indicated.

Complete locking solution using the fcntl module:
import fcntl
pid_file = 'program.pid'
fp = open(pid_file, 'w')
try:
fcntl.lockf(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
# another instance is running
sys.exit(1)

There are several common techniques including using semaphores. The one I see used most often is to create a "pid lock file" on startup that contains the pid of the running process. If the file already exists when the program starts up, open it up and grab the pid inside, check to see if a process with that pid is running, if it is check the cmdline value in /proc/pid to see if it is an instance of your program, if it is then quit, otherwise overwrite the file with your pid. The usual name for the pid file is application_name.pid.

wxWidgets offers a wxSingleInstanceChecker class for this purpose: wxPython doc, or wxWidgets doc. The wxWidgets doc has sample code in C++, but the python equivalent should be something like this (untested):
name = "MyApp-%s" % wx.GetUserId()
checker = wx.SingleInstanceChecker(name)
if checker.IsAnotherRunning():
return False

This builds upon the answer by user zgoda. It mainly addresses a tricky concern having to do with write access to the lock file. In particular, if the lock file was first created by root, another user foo can then no successfully longer attempt to rewrite this file due to an absence of write permissions for user foo. The obvious solution seems to be to create the file with write permissions for everyone. This solution also builds upon a different answer by me, having to do creating a file with such custom permissions. This concern is important in the real world where your program may be run by any user including root.
import fcntl, os, stat, tempfile
app_name = 'myapp' # <-- Customize this value
# Establish lock file settings
lf_name = '.{}.lock'.format(app_name)
lf_path = os.path.join(tempfile.gettempdir(), lf_name)
lf_flags = os.O_WRONLY | os.O_CREAT
lf_mode = stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH # This is 0o222, i.e. 146
# Create lock file
# Regarding umask, see https://stackoverflow.com/a/15015748/832230
umask_original = os.umask(0)
try:
lf_fd = os.open(lf_path, lf_flags, lf_mode)
finally:
os.umask(umask_original)
# Try locking the file
try:
fcntl.lockf(lf_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
msg = ('Error: {} may already be running. Only one instance of it '
'can run at a time.'
).format('appname')
exit(msg)
A limitation of the above code is that if the lock file already existed with unexpected permissions, those permissions will not be corrected.
I would've liked to use /var/run/<appname>/ as the directory for the lock file, but creating this directory requires root permissions. You can make your own decision for which directory to use.
Note that there is no need to open a file handle to the lock file.

Here's the TCP port-based solution:
# Use a listening socket as a mutex against multiple invocations
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('127.0.0.1', 5080))
s.listen(1)

Look for a python module that interfaces to SYSV semaphores on unix. The semaphores have a SEM_UNDO flag which will cause the resources held by the a process to be released if the process crashes.
Otherwise as Bernard suggested, you can use
import os
os.getpid()
And write it to /var/run/application_name.pid. When the process starts, it should check if the pid in /var/run/application_name.pid is listed in the ps table and quit if it is, otherwise write its own pid into /var/run/application_name.pid. In the following var_run_pid is the pid you read from /var/run/application_name.pid
cmd = "ps -p %s -o comm=" % var_run_pid
app_name = os.popen(cmd).read().strip()
if len(app_name) > 0:
Already running

The set of functions defined in semaphore.h -- sem_open(), sem_trywait(), etc -- are the POSIX equivalent, I believe.

If you create a lock file and put the pid in it, you can check your process id against it and tell if you crashed, no?
I haven't done this personally, so take with appropriate amounts of salt. :p

Can you use the 'pidof' utility? If your app is running, pidof will write the Process ID of your app to stdout. If not, it will print a newline (LF) and return an error code.
Example (from bash, for simplicity):
linux# pidof myapp
8947
linux# pidof nonexistent_app
linux#

By far the most common method is to drop a file into /var/run/ called [application].pid which contains only the PID of the running process, or parent process.
As an alternative, you can create a named pipe in the same directory to be able to send messages to the active process, e.g. to open a new file.

I've made a basic framework for running these kinds of applications when you want to be able to pass the command line arguments of subsequent attempted instances to the first one. An instance will start listening on a predefined port if it does not find an instance already listening there. If an instance already exists, it sends its command line arguments over the socket and exits.
code w/ explanation