Develop Reference

How to run a background process and do *not* wait?

My goal is simple: kick off rsync and DO NOT WAIT.
Python 2.7.9 on Debian
Sample code:
rsync_cmd = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}#{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1)
rsync_cmd2 = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}#{1}:'{2}' {3} &".format(remote_user, remote_server, file1, file1)
rsync_path = "/usr/bin/rsync"
rsync_args = shlex.split("-a -e 'ssh -i /home/mysuser/.ssh/id_rsa' {0}#{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1))
#subprocess.call(rsync_cmd, shell=True) # This isn't supposed to work but I tried it
#subprocess.Popen(rsync_cmd, shell=True) # This is supposed to be the solution but not for me
#subprocess.Popen(rsync_cmd2, shell=True) # Adding my own shell "&" to background it, still fails
#subprocess.Popen(rsync_cmd, shell=True, stdin=None, stdout=None, stderr=None, close_fds=True) # This doesn't work
#subprocess.Popen(shlex.split(rsync_cmd)) # This doesn't work
#os.execv(rsync_path, rsync_args) # This doesn't work
#os.spawnv(os.P_NOWAIT, rsync_path, rsync_args) # This doesn't work
#os.system(rsync_cmd2) # This doesn't work
print "DONE"
(I've commented out the execution commands only because I'm actually keeping all of my trials in my code so that I know what I've done and what I haven't done. Obviously, I would run the script with the right line uncommented.)
What happens is this...I can watch the transfer on the server and when it's finished, then I get a "DONE" printed to the screen.
What I'd like to have happen is a "DONE" printed immediately after issuing the rsync command and for the transfer to start.
Seems very straight-forward. I've followed details outlined in other posts, like this one and this one, but something is preventing it from working for me.
Thanks ahead of time.
(I have tried everything I can find in StackExchange and don't feel like this is a duplicate because I still can't get it to work. Something isn't right in my setup and need help.)

Here is verified example for Python REPL:
>>> import subprocess
>>> import sys
>>> p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(100)'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT); print('finished')
finished
How to verify that via another terminal window:
$ ps aux | grep python
Output:
user 32820 0.0 0.0 2447684 3972 s003 S+ 10:11PM 0:00.01 /Users/user/venv/bin/python -c import time; time.sleep(100)

Popen() starts a child process—it does not wait for it to exit. You have to call .wait() method explicitly if you want to wait for the child process. In that sense, all subprocesses are background processes.
On the other hand, the child process may inherit various properties/resources from the parent such as open file descriptors, the process group, its control terminal, some signal configuration, etc—it may lead to preventing ancestors processes to exit e.g., Python subprocess .check_call vs .check_output or the child may die prematurely on Ctrl-C (SIGINT signal is sent to the foreground process group) or if the terminal session is closed (SIGHUP).
To disassociate the child process completely, you should make it a daemon. Sometimes something in between could be enough e.g., it is enough to redirect the inherited stdout in a grandchild so that .communicate() in the parent would return when its immediate child exits.

I encountered a similar issue while working with qnx devices and wanted a sub-process that runs independently of the main process and even runs after the main process terminates.
Here's the solution I found that actually works 'creationflags=subprocess.DETACHED_PROCESS':
import subprocess
import time
pid = subprocess.Popen(["python", "path_to_script\turn_ecu_on.py"], creationflags=subprocess.DETACHED_PROCESS)
time.sleep(15)
print("Done")
Link to the doc: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

In Ubuntu the following commands keep working even if python app exits.
url = "https://www.youtube.com/watch?v=t3kcqTE6x4A"
cmd = f"mpv '{url}' && zenity --info --text 'you have watched {url}' &"
os.system(cmd)

python forked processes not executing with os.execlp

I've got this simple python script that ought to fork new processes and then have each execute a command using os.execlp, but the execution only occurs once. I'm curious if there's a timing issue going on that is preventing the additional forks from executing:
import os
for n in range(5):
PID = os.fork()
if PID == 0: #the child...
print("This child's PID is: %s" % os.getpid())
os.execlp('open','-n','-a','Calculator')
# os.popen('open -n -a Calculator')
# os._exit(0)
else:
print("new child forked: %d" % PID)
For this, the "open -n -a Appname" command in OS X launches a new instance of the specified application, so the above code should replace the forked process with the "open" command, and this should run 5 times. However, it only runs once so only one instance of Calculator is opened. Despite this, the parent lists 5 child PIDs forked.
If I comment out the os.execlp line and uncomment the os.popen and os._exit lines following it, then this works properly and the child processes all run the "open" command; however, I am wondering why the approach of replacing the forked process using execlp (or execvp and other similar variants) is not working? Clearly the child process is running, as I can use piping to run the "open" command just fine.
This is with python 3.4.3.

The first argument, after the executable, is arg[0] which is by convention the name of the executable. This is useful, if you have symbolic links, which determine the behavior of the programm. In your case, you name the programm '-n' and the real arguments are only -a and Calculator. So you have to repeat 'open':
os.execlp('open', 'open', '-n', '-a', 'Calculator')

How do I fork a new process with independent stdout, stderr, and stdin?

I have read most of the related questions about subprocess and os.fork(), including all the discussions about the double forking trick. However, none of the those solutions appear to work correctly for my scenario.
I want to fork a new process and allow the parent to terminate (normally) without screwing up the child's stdin, stdout, and stderr and without killing the child.
My first attempt was to use subprocess.Popen().
#!/usr/bin/python
from subprocess import call,Popen
Popen("/bin/bash", shell=True)
call("echo Hello > /tmp/FooBar", shell=True)
This fails because the child process is killed once the parent is exits. I am aware of creationflags but that is Windows-specific and I am running on Linux. Note that the above code works beautifully if we simply keep the parent process alive by adding an infinite loop to the end of it. This is undesirable because the parent is already finished with its job and there's no real reason for it to stick around.
The second attempt was to use os.fork().
#!/usr/bin/python
from subprocess import call
from os import fork
try:
pid = fork()
if pid > 0:
pass
else: # child process will start interactive process
call("/bin/bash", shell=True)
except:
print "Forking failed!"
call("echo Hello > /tmp/FooBar", shell=True)
Here, the child process no longer dies with the parent, but after the parent's death the child can no longer read input and write output.
Thus, I want to know how I fork a new process with utterly independent stdout, stderr, and stdin. Independence means that the parent process can terminate (normally), and the child process (whether it is bash or tmux or any other interactive program) behaves exactly as though the parent program had not terminated. To be even more precise, consider the following variation of the original program.
#!/usr/bin/python
from subprocess import call,Popen
Popen("/bin/bash", shell=True)
call("echo Hello > /tmp/FooBar", shell=True)
while True:
pass
The above code has all the behaviors I seek, but it keeps the Python process alive artificially. I am trying to achieve this exact behavior, without the Python process being alive.
Caveat: I am running these applications over ssh, so spawning a new GUI window is not a viable solution.
Desired Behavior:
I run the python code.
I get a shiny new bash shell that works exactly like the bash shell I started with.
The file /tmp/FooBar is created.
The original Python script finishes.
I continue on with my shiny new bash shell, and the output of ps aux | grep python does not include python script I just ran.

Your first example has an extra unnecessary call to Popen(), as the call convenience function will just execute its commands in a shell and exit, so it would work if you just ran:
from subprocess import call, Popen
call("echo Hello > /tmp/FooBar", shell=True)
However if you want to send a series of commands to a shell then the shell needs to opened with stdin attached to pipe so it doesn't get mixed up with the parents stdin (which is effectively what you're asking for in terms of obtaining independent stdio):
p = Popen("/bin/bash", shell=True, stdin = subprocess.PIPE)
p.stdin.write("echo Hello > /tmp/FooBar\n")
p.stdin.write("date >> /tmp/FooBar\n")
p.terminate()
If you want to also control the output then you redirect stdout and stderr to PIPE (i.e. stdout = subprocess.PIPE, stderr = subprocess.PIPE) and then call p.stdout.read() to obtain output as needed.
To allow the process to continue running after exiting python one can add & operator to the end of the command so it continues running in the background e.g.:
call("nc -l 2000 > /tmp/nc < &",shell=True)
To have a process run so that both stdin and stdout and still connected one can use shell redirects. To maintain access to stdin one can create a named pipe using mkfifo e.g:
call("mkfifo /tmp/pipe",shell=True)
call("tail -f /tmp/pipe > /tmp/out &",shell=True)
To provide input stdin one just send data to the pipe .e.g. from the shell:
$ echo 'test' > /tmp/pipe

I recently encountered this problem, I found a solution that might help,
using creationflags to tell Popen that the child process should not inherit the parent process console, and thus has it's own stdout, stdin and stderr as if it was a parent process.
subprocess.Popen("/bin/bash", creationflags= subprocess.CREATE_NEW_CONSOLE)
Popen supports creationflags keyword argument according to docs:
creationflags, if given, can be one or more of the following flags:
CREATE_NEW_CONSOLE
CREATE_NEW_PROCESS_GROUP
ABOVE_NORMAL_PRIORITY_CLASS
BELOW_NORMAL_PRIORITY_CLASS
HIGH_PRIORITY_CLASS
IDLE_PRIORITY_CLASS
NORMAL_PRIORITY_CLASS
REALTIME_PRIORITY_CLASS
CREATE_NO_WINDOW
DETACHED_PROCESS
CREATE_DEFAULT_ERROR_MODE
CREATE_BREAKAWAY_FROM_JOB
The ones that you are interested in are DETACHED_PROCESS and CREATE_NEW_CONSOLE, I've used CREATE_NEW_CONSOLE, what it does is it spawns a new process as if it was a parent, I used Python3.5 Windows, in Python 3.7 DETACHED_PROCESS was added and it's documented to do the same as CREATE_NEW_CONSOLE.

how to stop the current execution of a system call in python

From python I am calling a java function:
os.system("java -jar example.jar run myFunction 'inFile.txt' 'outFile.txt' " )
This function is processing a file and the output is written into 'outFile.txt'. The output is dependent on the information in 'inFile.txt'. While processing the input file and writing into out file, sometimes the 'outFile.txt' grows too large (tens of GBs) and at that time, i want to quit and the current processing and move on to process another inFile.txt
Is there is way to know that my outFile.txt that is being written has grown more than say 10GB.
Edit:
As suggested by Maksym, I am using the following code and seems to be working. Thanks
import subprocess
from time import sleep
p = subprocess.Popen(["java", "-jar", "example.jar", "run", "myFunction", "'inFile.txt'", "'outFile.txt'")
rc = p.poll() #returncode
while (rc == None):
sleep(1)
if(os.path.getsize(outFileName) < 1000000000):
rc = p.poll()
continue
else:
p.kill()
break

Have a look at subprocess module. Using Popen you can fork a process and kill it when you need this:
import subprocess
from time import sleep
p = subprocess.Popen(["java", "-jar", "example.jar", "run", "myFunction", "'inFile.txt'", "'outFile.txt'")
while not check_my_conditions():
sleep(my_timeout)
p.kill()
Then, you can rotate your files and restart the process.

Instead of directly calling os.system, you should strongly considering using the multiprocessing.Process built-in class. It handles dealing with spawned processes much more gracefully.
You need to watch the output file periodically, either using something like os.stat to check the file size. You can then kill the original process (or whatever you want to do) when the threshold is exceeded.

Does the java application provide any output (for example, a count of records processed) to stdout or stderr while it runs? If so, you could invoke it using Python's Popen class (in the subprocess module) and estimate when it has processed 'too much'.

python run multi command in the same time

Prior to this,I run two command in for loop,like
for x in $set:
command
In order to save time,i want to run these two command in the same time,like parallel method in makefile
Thanks
Lyn

The threading module won't give you much performance-wise because of the Global Interpreter Lock.
I think the best way to do this is to use the subprocess module and open each command with it's own stdout.
processes = {}
for cmd in ['cmd1', 'cmd2', 'cmd3']:
p = subprocess.Popen('cmd1', stdout=subprocess.PIPE)
processes[p.stdout] = p
while len(processes):
rfds, _, _ = select.select(processes.keys(), [], [])
for fd in rfds:
process = processses[fd]
print fd.read()
if process.returncode is not None:
print "Process {0} returned with code {1}".format(process.pid, process.returncode)
del processes[fd]
You basically have to use select to see which file descriptors are ready and you have to check their returncode to see if doing a "read" caused them to exit. Processes basically go into a wait state until their stdout is closed. If you would like to do some things while you're waiting, you can put a timeout on select.select() so you'll stop waiting after so long. You can test the length of rfds and if it is 0 then you know that the timeout happened.

twisted or select module is probably what you're after.
If all you want to do is a bunch of batch commands, shell scripts, ie
#!/bin/sh
for i in "command1 command2 command3"; do
$i &
done
Might work better. Alternately, a Makefile like you said.

Look at the threading module.