I'm trying to port a shell script to the much more readable python version. The original shell script starts several processes (utilities, monitors, etc.) in the background with "&". How can I achieve the same effect in python? I'd like these processes not to die when the python scripts complete. I am sure it's related to the concept of a daemon somehow, but I couldn't find how to do this easily.
While jkp's solution works, the newer way of doing things (and the way the documentation recommends) is to use the subprocess module. For simple commands its equivalent, but it offers more options if you want to do something complicated.
Example for your case:
import subprocess
subprocess.Popen(["rm","-r","some.file"])
This will run rm -r some.file in the background. Note that calling .communicate() on the object returned from Popen will block until it completes, so don't do that if you want it to run in the background:
import subprocess
ls_output=subprocess.Popen(["sleep", "30"])
ls_output.communicate() # Will block for 30 seconds
See the documentation here.
Also, a point of clarification: "Background" as you use it here is purely a shell concept; technically, what you mean is that you want to spawn a process without blocking while you wait for it to complete. However, I've used "background" here to refer to shell-background-like behavior.
Note: This answer is less current than it was when posted in 2009. Using the subprocess module shown in other answers is now recommended in the docs
(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)
If you want your process to start in the background you can either use system() and call it in the same way your shell script did, or you can spawn it:
import os
os.spawnl(os.P_DETACH, 'some_long_running_command')
(or, alternatively, you may try the less portable os.P_NOWAIT flag).
See the documentation here.
You probably want the answer to "How to call an external command in Python".
The simplest approach is to use the os.system function, e.g.:
import os
os.system("some_command &")
Basically, whatever you pass to the system function will be executed the same as if you'd passed it to the shell in a script.
I found this here:
On windows (win xp), the parent process will not finish until the longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.
The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen([sys.executable, "longtask.py"],
creationflags=DETACHED_PROCESS).pid
Use subprocess.Popen() with the close_fds=True parameter, which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits.
https://gist.github.com/yinjimmy/d6ad0742d03d54518e9f
import os, time, sys, subprocess
if len(sys.argv) == 2:
time.sleep(5)
print 'track end'
if sys.platform == 'darwin':
subprocess.Popen(['say', 'hello'])
else:
print 'main begin'
subprocess.Popen(['python', os.path.realpath(__file__), '0'], close_fds=True)
print 'main end'
Both capture output and run on background with threading
As mentioned on this answer, if you capture the output with stdout= and then try to read(), then the process blocks.
However, there are cases where you need this. For example, I wanted to launch two processes that talk over a port between them, and save their stdout to a log file and stdout.
The threading module allows us to do that.
First, have a look at how to do the output redirection part alone in this question: Python Popen: Write to stdout AND log file simultaneously
Then:
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
import threading
def output_reader(proc, file):
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
file.buffer.write(byte)
else:
break
with subprocess.Popen(['./sleep.py', '0'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc1, \
subprocess.Popen(['./sleep.py', '10'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc2, \
open('log1.log', 'w') as file1, \
open('log2.log', 'w') as file2:
t1 = threading.Thread(target=output_reader, args=(proc1, file1))
t2 = threading.Thread(target=output_reader, args=(proc2, file2))
t1.start()
t2.start()
t1.join()
t2.join()
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(4):
print(i + int(sys.argv[1]))
sys.stdout.flush()
time.sleep(0.5)
After running:
./main.py
stdout get updated every 0.5 seconds for every two lines to contain:
0
10
1
11
2
12
3
13
and each log file contains the respective log for a given process.
Inspired by: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
Tested on Ubuntu 18.04, Python 3.6.7.
You probably want to start investigating the os module for forking different threads (by opening an interactive session and issuing help(os)). The relevant functions are fork and any of the exec ones. To give you an idea on how to start, put something like this in a function that performs the fork (the function needs to take a list or tuple 'args' as an argument that contains the program's name and its parameters; you may also want to define stdin, out and err for the new thread):
try:
pid = os.fork()
except OSError, e:
## some debug output
sys.exit(1)
if pid == 0:
## eventually use os.putenv(..) to set environment variables
## os.execv strips of args[0] for the arguments
os.execv(args[0], args)
You can use
import os
pid = os.fork()
if pid == 0:
Continue to other code ...
This will make the python process run in background.
I haven't tried this yet but using .pyw files instead of .py files should help. pyw files dosen't have a console so in theory it should not appear and work like a background process.
I've got a long running python script that I want to be able to end from another python script. Ideally what I'm looking for is some way of setting a process ID to the first script and being able to see if it is running or not via that ID from the second. Additionally, I'd like to be able to terminate that long running process.
Any cool shortcuts exist to make this happen?
Also, I'm working in a Windows environment.
I just recently found an alternative answer here: Check to see if python script is running
You could get your own PID (Process Identifier) through
import os
os.getpid()
and to kill a process in Unix
import os, signal
os.kill(5383, signal.SIGKILL)
to kill in Windows use
import subprocess as s
def killProcess(pid):
s.Popen('taskkill /F /PID {0}'.format(pid), shell=True)
You can send the PID to the other programm or you could search in the process-list to find the name of the other script and kill it with the above script.
I hope that helps you.
You're looking for the subprocess module.
import subprocess as sp
extProc = sp.Popen(['python','myPyScript.py']) # runs myPyScript.py
status = sp.Popen.poll(extProc) # status should be 'None'
sp.Popen.terminate(extProc) # closes the process
status = sp.Popen.poll(extProc) # status should now be something other than 'None' ('1' in my testing)
subprocess.Popen starts the external python script, equivalent to typing 'python myPyScript.py' in a console or terminal.
The status from subprocess.Popen.poll(extProc) will be 'None' if the process is still running, and (for me) 1 if it has been closed from within this script. Not sure about what the status is if it has been closed another way.
This worked for me under windows 11 and PyQt5:
subprocess.Popen('python3 MySecondApp.py')
Popen.terminate(app)
where app is MyFirstApp.py (the caller script, running) and MySecondApp.py (the called script)
This question already has answers here:
Using module 'subprocess' with timeout
(31 answers)
Closed 7 years ago.
I wrote a program (I ran it in the terminal) that goes through a list of terminal commands (Kali Linux).
import subprocess as sub
import time
sub.call(['airmon-ng', 'start', 'wlan0'])
p = sub.call(['airodump-ng','wlan0mon'])
time.sleep(10)
p.kill()
The last commmand is airodump-ng wlan0mon. Everything works fine (everything is displayed in the terminal (beacons, essid, etc.).
After a specified time I wish to kill the process (airodump-ng wlan0mon).
I don't want to press Ctrl + C by hand!
p.kill() does not work (maybe improper use).
How can I do this? What command should I send through the subprocess module?
The subprocess.call method is a high-level API that waits for the process to terminate before returning its exit code. If you need your main process to continue running while the subprocess runs, you need to use the slightly lower-level API: subprocess.Popen, which starts the process in the background.
Using p = sub.Popen(['airodump-ng','wlan0mon']) instead of p = sub.call(['airodump-ng','wlan0mon']) should work.
I am attempting to to launch a python script from within another python script, but in a minimized console, then return control to the original shell.
I am able to open the required script in a new shell below, but it's not minimized:
#!/usr/bin/env python
import os
import sys
import subprocess
pyTivoPath="c:\pyTivo\pyTivo.py"
print "Testing: Open New Console"
subprocess.Popen([sys.executable, pyTivoPath], creationflags = subprocess.CREATE_NEW_CONSOLE)
print
raw_input("Press Enter to continue...")
Further, I will need to be able to later remotely KILL this shell from the original script, so I suspect I'll need to be explicit in naming the new process. Correct?
Looking for pointers, please. Thanks!
Note: python27 is mandatory for this application. Eventually will also need to work on Mac and Linux.
Do you need to have the other console open? If you now the commands to be sent, then I'd recommend using Popen.communicate(input="Shell commands") and it will automate the process for you.
So you could write something along the lines of:
# Commands to pass into subprocess (each command is separated by a newline)
commands = (
"command1\n" +
"command2\n"
)
# Your process
py_process = subprocess.Popen(*yourprocess_here*, stdin=PIPE, shell=True)
# Feed process the needed input
py_process.communicate(input=commands)
# Terminate when finished
py_process.terminate()
The code above will execute the process you specify and even send commands but it won't open a new console.
I need to run multiple programs one after the other and they each run in a console window.
I want the console window to be visible, but a new window is created for each program. This is annoying because each window is opened in a new position from where the other is closed and steals focus when working in Eclipse.
This is the initial code I was using:
def runCommand( self, cmd, instream=None, outstream=None, errstream=None ):
proc = subprocess.Popen( cmd, stdin=instream, stdout=outstream, stderr=errstream )
while True:
retcode = proc.poll()
if retcode == None:
if mAbortBuild:
proc.terminate()
return False
else:
time.sleep(1)
else:
if retcode == 0:
return True
else:
return False
I switched to opening a command prompt using 'cmd' when calling subprocess.Popen and then calling proc.stdin.write( b'program.exe\r\n' ).
This seems to solve the one command window problem but now I can't tell when the first program is done and I can start the second. I want to stop and interrogate the log file from the first program before running the second program.
Any tips on how I can achieve this? Is there another option for running the programs in one window I haven't found yet?
Since you're using Windows, you could just create a batch file listing each program you want to run which will all execute in a single console window. Since it's a batch script you can do things like put conditional statements in it as shown in the example.
import os
import subprocess
import textwrap
# create a batch file with some commands in it
batch_filename = 'commands.bat'
with open(batch_filename, "wt") as batchfile:
batchfile.write(textwrap.dedent("""
python hello.py
if errorlevel 1 (
#echo non-zero exit code: %errorlevel% - terminating
exit
)
time /t
date /t
"""))
# execute the batch file as a separate process and echo its output
kwargs = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
with subprocess.Popen(batch_filename, **kwargs).stdout as output:
for line in output:
print line,
try: os.remove(batch_filename) # clean up
except os.error: pass
In section 17.5.3.1. Constants in the subprocess module documentation there's description of subprocess.CREATE_NEW_CONSOLE constant:
The new process has a new console, instead of inheriting its parent’s
console (the default).
As we see, by default, new process inherits its parent's console. The reason you observe multiple consoles being opened is the fact that you call your scripts from within Eclipse, which itself does not have console so each subprocess creates its own console as there's no console it could inherit. If someone would like to simulate this behavior it's enough to run Python script which creates subprocesses using pythonw.exe instead of python.exe. The difference between the two is that the former does not open a console whereas the latter does.
The solution is to have helper script — let's call it launcher — which, by default, creates console and runs your programs in subprocesses. This way each program inherits one and the same console from its parent — the launcher. To run programs sequentially we use Popen.wait() method.
--- script_run_from_eclipse.py ---
import subprocess
import sys
subprocess.Popen([sys.executable, 'helper.py'])
--- helper.py ---
import subprocess
programs = ['first_program.exe', 'second_program.exe']
for program in programs:
subprocess.Popen([program]).wait()
if input('Do you want to continue? (y/n): ').upper() == 'N':
break