Subprocess Popen problem with terminate shell script - python

I have a little problem with understand what is going on with my code.
I have a lot of lines in my code so i decided to simulate my problem with shorter version.
Im using raspberry pi 4, flirone on rapsbian.
import sys
import os
import time
import subprocess
global shellscript
#global pid
def subprocess_settings():
shellscript = subprocess.Popen(["/home/pi/Desktop/init_flir.sh"], close_fds=True)
#shellscript = subprocess.Popen(["/home/pi/Desktop/init_flir.sh", "&> /dev/null"], stdin=None, stdout=None, close_fds=True)
#pid = shellscript.pid
try:
x = 1
while True:
if x == 0:
sys.exit()
elif x == 1:
print("Yas, x=1")
time.sleep(2)
subprocess_settings()
time.sleep(5)
print("Driver test is in background mode")
time.sleep(2)
print("Close process")
subprocess.Popen.kill(shellscript)
x=0
except KeyboardInterrupt:
subprocess.Popen.kill(shellscript)
and my .sh file:
#!/bin/bash
cd /home/pi/github/flirone-v4l2
sudo modprobe v4l2loopback exclusive_caps=0,0 video_nr=1,2,3
sudo ./flirone ~/github/flirone-v4l2/palettes/Iron2.raw
I want to run my .sh file. It has to work in background. I need to terminate this with 2 ways, first one from keyboard and second inside my elif.
At the end i need to hide terminal output from .sh file.
What im doing wrong??
Is this is a problem with global variables?
How to fix this to work properly?
Thanks for any help.

This solution works properly, as i wanted. In flirone driver i runned ./flirone and opened 1 process from subprocess.Popen and 2 additional processes in flirone code. I checked this running command sudo ./flirone ~/github/flirone-v4l2/palettes/Iron2.raw and ps aux | grep flirone . I used pidof flirone command to check main ID. When im killing this process everything works as should be and other processes ends too.
import sys
import os
import signal
import time
import subprocess
def subprocess_kill():
pidof = subprocess.check_output("pidof flirone", shell=True)
time.sleep(1)
subprocess.check_output("sudo kill -9 %s " % (pidof), shell=True)
try:
x = 1
while True:
if x == 0:
sys.exit()
elif x == 1:
print("x=1")
time.sleep(2)
os.chdir("/home/pi/github/flirone-v4l2")
shellscript = subprocess.Popen("sudo modprobe v4l2loopback exclusive_caps=0,0 video_nr=1,2,3", shell=True)
shellscript2 = subprocess.Popen("sudo ./flirone ~/github/flirone-v4l2/palettes/Iron2.raw", shell=True)
time.sleep(5)
print("Driver test is in background mode")
time.sleep(2)
print("Close process")
subprocess_kill()
x=0
except KeyboardInterrupt:
print("hey")
When i deleted killing method from except KeyboardInterrupt processes ends too.
If you want to hide output of subprocess u can use:
FNULL = open(os.devnull, "w")
shellscript2 = subprocess.Popen("sudo ./flirone ~/github/flirone-v4l2/palettes/Iron2.raw", shell=True, stdout=FNULL)
More: How to hide output of subprocess in Python 2.7
Subprocess python 2.7 lib

Related

Unable to prevent program to print a Error

I took a piece of code from here:
https://stackoverflow.com/a/10457565/13882705
which is
# I have used os comands for a while
# this program will try to close a firefox window every ten secounds
import os
import time
# creating a forever loop
while 1 :
os.system("TASKKILL /F /IM firefox.exe")
time.sleep(10)
It will terminate a process if it is running using OS module
But if the program did not find the app we mentioned then it prints
ERROR: The process "firefox.exe" not found.
Is there a way to make the program just print application not found once and wait until the program is rerunned?
It is fine even if it just prints "Application Not found"
Use subprocess.run instead of os.system so you have more control:
import subprocess
import time
while True:
proc = subprocess.run(["TASKKILL", "/F", "/IM", "firefox.exe"], stderr=subprocess.PIPE)
if proc.returncode != 0:
print("Application not found.")
break # since application isn't here we just exit
time.sleep(10)

Stopping a subprocess in python

I have several python scripts that run at the same time with the use of the subprocess library. However, i want to be able to stop all processes when an event happens, such as the press of a button. I have tried several things, but none of them seem to work.
I'm using Windows with Python 3.8.0
Below is my code that opens all of the processes.
import subprocess
subprocess.Popen("python Program1.py")
subprocess.Popen("python Program2.py")
subprocess.Popen("python Program3.py")
subprocess.Popen("python Program4.py")
subprocess.Popen("python Program5.py")
subprocess.Popen("python Program6.py")
subprocess.Popen("python Program7.py")
I cant figure out how to end the processes with the subprocess library or any other ones.
I have also tried the following
proc1.kill()
p.terminate()
def pkill (process_name):
try:
killed = os.system('tskill ' + process_name)
except Exception as e:
killed = 0
return killed
pkill("PROGRAM")
And with the one above, i have tried putting except Exception, e: And also putting pkill("PROGRAM.py") but that doesnt seem to work either.
You could keep track of the launched Process Ids:
import os
import subprocess
processes = []
for x in range(0,9):
p = subprocess.Popen("python Program.py %s" % x)
processes.append(p.pid) # keep the processIds!
for p in processes:
os.system('taskkill /F /PID ' + p)

How to make gnome-terminal command act like object in python

I'm trying to create a gnome-terminal windows and after that I want to write something at terminal window (echo) like a buffering screen in a while loop.
import os
x = os.system("gnome-terminal -e")
while True:
x.write("echo % s "%(buffering))
if progress == 0:
break
So, I have to open gnome-terminal window before while loop because if I don't gnome terminal opens and close at each loop.
You could use a named pipe:
import os,time
def client():
try:
os.mkfifo("named_pipe")
except OSError as e:
print(e.message)
named_pipe = os.open("named_pipe", os.O_WRONLY)
while True:
time.sleep(1)
os.write(named_pipe, 'Time {}\n'.format(time.asctime()))
def server():
from subprocess import check_call
check_call(['gnome-terminal', '-e', "python script.py"])
pid = os.fork()
foo() if pid != 0 else bar()
In script.py:
import os
with open('named_pipe') as f:
for line in iter(f.readline, ""):
print(line.rstrip())
os.unlink("named_pipe")
You could also use a unix domain socket or tcp.

Python kill a subprocess(that starts another process) and start it again

I'm trying to make a python script that starts the program livestreamer (that starts the program mplayer) and after 10 seconds it should kill the program, or the subprocess. here is my current code that doesn't work, I think I know why but I don't know how to solve it.
I think the problem is that the subprocess starts livestreamer and then the program livestreamer starts the program mplayer. Python doesn't know about mplayer and can't close it. How would I be able to kill both livestreamer and mplayer after 10 second and then start them again as a loop?
I'm using Ubuntu 14.04 (Linux) and Python 2.7.6
import subprocess
import time
import os
import sys
import signal
url = "http://new.livestream.com/accounts/398160/events/3155348"
home = os.environ['HOME']
if not os.geteuid() == 0:
if not os.path.exists('/%s/.config/livestreamer' % home):
os.makedirs('/%s/.config/livestreamer' % home)
lscfg = open('%s/.config/livestreamer/config' % home, 'w+')
lscfg.write("player=mplayer -geometry 0%:0% -nomouseinput -loop 100 -noborder -fixed-vo")
lscfg.close()
cmd = "livestreamer %s best --player-continuous-http --player-no-close" % url
while True:
proc1 = subprocess.Popen(cmd.split(), shell=False)
time.sleep(10)
proc1.kill()
Solution:
import subprocess
import time
import os
import sys
import signal
url = "http://new.livestream.com/accounts/398160/events/3155348"
home = os.environ['HOME']
if not os.geteuid() == 0:
if not os.path.exists('/%s/.config/livestreamer' % home):
os.makedirs('/%s/.config/livestreamer' % home)
lscfg = open('%s/.config/livestreamer/config' % home, 'w+')
lscfg.write("player=mplayer -geometry 0%:0% -nomouseinput -loop 100 -noborder -fixed-vo")
lscfg.close()
cmd = "livestreamer %s best --player-continuous-http --player-no-close" % url
#restarting the player every 10th minute to catch up on possible delay
while True:
proc1 = subprocess.Popen(cmd.split(), shell=False)
time.sleep(600)
os.system("killall -9 mplayer")
proc1.kill()
As you can see os.system("killall -9 mplayer") was the command to kill the process mplayer.
In your code you kill livestreamer but not mplayer so mplayer will continue running.
By using kill on your subprocess you send a signal SIGKILL and unless the subprocess do handle the signal interruption it will simply close itself fast and without killing his own childs so mplayer will live (and may become a zombie process).
You have no reference to your subprocess child 'mplayer' but if you can get his PID you can kill it with os.kill(...)
os.kill(process_pid, signal.SIGTERM)
Using os.system("killall -9 mplayer") was the easy way to solve this. Mind using this option will kill all process of mplayer though this is not a problem in my case but may be a problem for other cases.
while True:
proc1 = subprocess.Popen(cmd.split(), shell=False)
time.sleep(600)
os.system("killall -9 mplayer")
proc1.kill()

Sending ^C to Python subprocess objects on Windows

I have a test harness (written in Python) that needs to shut down the program under test (written in C) by sending it ^C. On Unix,
proc.send_signal(signal.SIGINT)
works perfectly. On Windows, that throws an error ("signal 2 is not supported" or something like that). I am using Python 2.7 for Windows, so I have the impression that I should be able to do instead
proc.send_signal(signal.CTRL_C_EVENT)
but this doesn't do anything at all. What do I have to do? This is the code that creates the subprocess:
# Windows needs an extra argument passed to subprocess.Popen,
# but the constant isn't defined on Unix.
try: kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
except AttributeError: pass
proc = subprocess.Popen(argv,
stdin=open(os.path.devnull, "r"),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
**kwargs)
There is a solution by using a wrapper (as described in the link Vinay provided) which is started in a new console window with the Windows start command.
Code of the wrapper:
#wrapper.py
import subprocess, time, signal, sys, os
def signal_handler(signal, frame):
time.sleep(1)
print 'Ctrl+C received in wrapper.py'
signal.signal(signal.SIGINT, signal_handler)
print "wrapper.py started"
subprocess.Popen("python demo.py")
time.sleep(3) #Replace with your IPC code here, which waits on a fire CTRL-C request
os.kill(signal.CTRL_C_EVENT, 0)
Code of the program catching CTRL-C:
#demo.py
import signal, sys, time
def signal_handler(signal, frame):
print 'Ctrl+C received in demo.py'
time.sleep(1)
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print 'demo.py started'
#signal.pause() # does not work under Windows
while(True):
time.sleep(1)
Launch the wrapper like e.g.:
PythonPrompt> import subprocess
PythonPrompt> subprocess.Popen("start python wrapper.py", shell=True)
You need to add some IPC code which allows you to control the wrapper firing the os.kill(signal.CTRL_C_EVENT, 0) command. I used sockets for this purpose in my application.
Explanation:
Preinformation
send_signal(CTRL_C_EVENT) does not work because CTRL_C_EVENT is only for os.kill. [REF1]
os.kill(CTRL_C_EVENT) sends the signal to all processes running in the current cmd window [REF2]
Popen(..., creationflags=CREATE_NEW_PROCESS_GROUP) does not work because CTRL_C_EVENT is ignored for process groups. [REF2]
This is a bug in the python documentation [REF3]
Implemented solution
Let your program run in a different cmd window with the Windows shell command start.
Add a CTRL-C request wrapper between your control application and the application which should get the CTRL-C signal. The wrapper will run in the same cmd window as the application which should get the CTRL-C signal.
The wrapper will shutdown itself and the program which should get the CTRL-C signal by sending all processes in the cmd window the CTRL_C_EVENT.
The control program should be able to request the wrapper to send the CTRL-C signal. This might be implemnted trough IPC means, e.g. sockets.
Helpful posts were:
I had to remove the http in front of the links because I'm a new user and are not allowed to post more than two links.
http://social.msdn.microsoft.com/Forums/en-US/windowsgeneraldevelopmentissues/thread/dc9586ab-1ee8-41aa-a775-cf4828ac1239/#6589714f-12a7-447e-b214-27372f31ca11
Can I send a ctrl-C (SIGINT) to an application on Windows?
Sending SIGINT to a subprocess of python
http://bugs.python.org/issue9524
http://ss64.com/nt/start.html
http://objectmix.com/python/387639-sending-cntrl-c.html#post1443948
Update: IPC based CTRL-C Wrapper
Here you can find a selfwritten python module providing a CTRL-C wrapping including a socket based IPC.
The syntax is quite similiar to the subprocess module.
Usage:
>>> import winctrlc
>>> p1 = winctrlc.Popen("python demo.py")
>>> p2 = winctrlc.Popen("python demo.py")
>>> p3 = winctrlc.Popen("python demo.py")
>>> p2.send_ctrl_c()
>>> p1.send_ctrl_c()
>>> p3.send_ctrl_c()
Code
import socket
import subprocess
import time
import random
import signal, os, sys
class Popen:
_port = random.randint(10000, 50000)
_connection = ''
def _start_ctrl_c_wrapper(self, cmd):
cmd_str = "start \"\" python winctrlc.py "+"\""+cmd+"\""+" "+str(self._port)
subprocess.Popen(cmd_str, shell=True)
def _create_connection(self):
self._connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._connection.connect(('localhost', self._port))
def send_ctrl_c(self):
self._connection.send(Wrapper.TERMINATION_REQ)
self._connection.close()
def __init__(self, cmd):
self._start_ctrl_c_wrapper(cmd)
self._create_connection()
class Wrapper:
TERMINATION_REQ = "Terminate with CTRL-C"
def _create_connection(self, port):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost', port))
s.listen(1)
conn, addr = s.accept()
return conn
def _wait_on_ctrl_c_request(self, conn):
while True:
data = conn.recv(1024)
if data == self.TERMINATION_REQ:
ctrl_c_received = True
break
else:
ctrl_c_received = False
return ctrl_c_received
def _cleanup_and_fire_ctrl_c(self, conn):
conn.close()
os.kill(signal.CTRL_C_EVENT, 0)
def _signal_handler(self, signal, frame):
time.sleep(1)
sys.exit(0)
def __init__(self, cmd, port):
signal.signal(signal.SIGINT, self._signal_handler)
subprocess.Popen(cmd)
conn = self._create_connection(port)
ctrl_c_req_received = self._wait_on_ctrl_c_request(conn)
if ctrl_c_req_received:
self._cleanup_and_fire_ctrl_c(conn)
else:
sys.exit(0)
if __name__ == "__main__":
command_string = sys.argv[1]
port_no = int(sys.argv[2])
Wrapper(command_string, port_no)
New answer:
When you create the process, use the flag CREATE_NEW_PROCESS_GROUP. And then you can send CTRL_BREAK to the child process. The default behavior is the same as CTRL_C, except that it won't affect the calling process.
Old answer:
My solution also involves a wrapper script, but it does not need IPC, so it is far simpler to use.
The wrapper script first detaches itself from any existing console, then attach to the target console, then files the Ctrl-C event.
import ctypes
import sys
kernel = ctypes.windll.kernel32
pid = int(sys.argv[1])
kernel.FreeConsole()
kernel.AttachConsole(pid)
kernel.SetConsoleCtrlHandler(None, 1)
kernel.GenerateConsoleCtrlEvent(0, 0)
sys.exit(0)
The initial process must be launched in a separate console so that the Ctrl-C event will not leak. Example
p = subprocess.Popen(['some_command'], creationflags=subprocess.CREATE_NEW_CONSOLE)
# Do something else
subprocess.check_call([sys.executable, 'ctrl_c.py', str(p.pid)]) # Send Ctrl-C
where I named the wrapper script as ctrl_c.py.
Try calling the GenerateConsoleCtrlEvent function using ctypes. As you are creating a new process group, the process group ID should be the same as the pid. So, something like
import ctypes
ctypes.windll.kernel32.GenerateConsoleCtrlEvent(0, proc.pid) # 0 => Ctrl-C
should work.
Update: You're right, I missed that part of the detail. Here's a post which suggests a possible solution, though it's a bit kludgy. More details are in this answer.
Here is a fully working example which doesn't need any modification in the target script.
This overrides the sitecustomize module so it might no be suitable for every scenario. However, in this case you could use a *.pth file in site-packages to execute code at the subprocess startup (see https://nedbatchelder.com/blog/201001/running_code_at_python_startup.html).
Edit This works only out of the box for subprocesses in Python. Other processes have to manually call SetConsoleCtrlHandler(NULL, FALSE).
main.py
import os
import signal
import subprocess
import sys
import time
def main():
env = os.environ.copy()
env['PYTHONPATH'] = '%s%s%s' % ('custom-site', os.pathsep,
env.get('PYTHONPATH', ''))
proc = subprocess.Popen(
[sys.executable, 'sub.py'],
env=env,
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
)
time.sleep(1)
proc.send_signal(signal.CTRL_C_EVENT)
proc.wait()
if __name__ == '__main__':
main()
custom-site\sitecustomize.py
import ctypes
import sys
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
if not kernel32.SetConsoleCtrlHandler(None, False):
print('SetConsoleCtrlHandler Error: ', ctypes.get_last_error(),
file=sys.stderr)
sub.py
import atexit
import time
def cleanup():
print ('cleanup')
atexit.register(cleanup)
while True:
time.sleep(1)
I have a single file solution with the following advantages:
- No external libraries. (Other than ctypes)
- Doesn't require the process to be opened in a specific way.
The solution is adapted from this stack overflow post, but I think it's much more elegant in python.
import os
import signal
import subprocess
import sys
import time
# Terminates a Windows console app sending Ctrl-C
def terminateConsole(processId: int, timeout: int = None) -> bool:
currentFilePath = os.path.abspath(__file__)
# Call the below code in a separate process. This is necessary due to the FreeConsole call.
try:
code = subprocess.call('{} {} {}'.format(sys.executable, currentFilePath, processId), timeout=timeout)
if code == 0: return True
except subprocess.TimeoutExpired:
pass
# Backup plan
subprocess.call('taskkill /F /PID {}'.format(processId))
if __name__ == '__main__':
pid = int(sys.argv[1])
import ctypes
kernel = ctypes.windll.kernel32
r = kernel.FreeConsole()
if r == 0: exit(-1)
r = kernel.AttachConsole(pid)
if r == 0: exit(-1)
r = kernel.SetConsoleCtrlHandler(None, True)
if r == 0: exit(-1)
r = kernel.GenerateConsoleCtrlEvent(0, 0)
if r == 0: exit(-1)
r = kernel.FreeConsole()
if r == 0: exit(-1)
# use tasklist to wait while the process is still alive.
while True:
time.sleep(1)
# We pass in stdin as PIPE because there currently is no Console, and stdin is currently invalid.
searchOutput: bytes = subprocess.check_output('tasklist /FI "PID eq {}"'.format(pid), stdin=subprocess.PIPE)
if str(pid) not in searchOutput.decode(): break;
# The following two commands are not needed since we're about to close this script.
# You can leave them here if you want to do more console operations.
r = kernel.SetConsoleCtrlHandler(None, False)
if r == 0: exit(-1)
r = kernel.AllocConsole()
if r == 0: exit(-1)
exit(0)
For those interested in a "quick fix", I've made a console-ctrl package based on Siyuan Ren's answer to make it even easier to use.
Simply run pip install console-ctrl, and in your code:
import console_ctrl
import subprocess
# Start some command IN A SEPARATE CONSOLE
p = subprocess.Popen(['some_command'], creationflags=subprocess.CREATE_NEW_CONSOLE)
# ...
# Stop the target process
console_ctrl.send_ctrl_c(p.pid)
I have been trying this but for some reason ctrl+break works, and ctrl+c does not. So using os.kill(signal.CTRL_C_EVENT, 0) fails, but doing os.kill(signal.CTRL_C_EVENT, 1) works. I am told this has something to do with the create process owner being the only one that can pass a ctrl c? Does that make sense?
To clarify, while running fio manually in a command window it appears to be running as expected. Using the CTRL + BREAK breaks without storing the log as expected and CTRL + C finishes writing to the file also as expected. The problem appears to be in the signal for the CTRL_C_EVENT.
It almost appears to be a bug in Python but may rather be a bug in Windows. Also one other thing, I had a cygwin version running and sending the ctrl+c in python there worked as well, but then again we aren't really running native windows there.
example:
import subprocess, time, signal, sys, os
command = '"C:\\Program Files\\fio\\fio.exe" --rw=randrw --bs=1M --numjobs=8 --iodepth=64 --direct=1 ' \
'--sync=0 --ioengine=windowsaio --name=test --loops=10000 ' \
'--size=99901800 --rwmixwrite=100 --do_verify=0 --filename=I\\:\\test ' \
'--thread --output=C:\\output.txt'
def signal_handler(signal, frame):
time.sleep(1)
print 'Ctrl+C received in wrapper.py'
signal.signal(signal.SIGINT, signal_handler)
print 'command Starting'
subprocess.Popen(command)
print 'command started'
time.sleep(15)
print 'Timeout Completed'
os.kill(signal.CTRL_C_EVENT, 0)
(This was supposed to be a comment under Siyuan Ren's answer but I don't have enough rep so here's a slightly longer version.)
If you don't want to create any helper scripts you can use:
p = subprocess.Popen(['some_command'], creationflags=subprocess.CREATE_NEW_CONSOLE)
# Do something else
subprocess.run([
sys.executable,
"-c",
"import ctypes, sys;"
"kernel = ctypes.windll.kernel32;"
"pid = int(sys.argv[1]);"
"kernel.FreeConsole();"
"kernel.AttachConsole(pid);"
"kernel.SetConsoleCtrlHandler(None, 1);"
"kernel.GenerateConsoleCtrlEvent(0, 0);"
"sys.exit(0)",
str(p.pid)
]) # Send Ctrl-C
But it won't work if you use PyInstaller - sys.executable points to your executable, not the Python interpreter. To solve that issue I've created a tiny utility for Windows: https://github.com/anadius/ctrlc
Now you can send the Ctrl+C event with:
subprocess.run(["ctrlc", str(p.pid)])

Categories

Resources