Develop Reference

How do I run multiple commands with python subprocess( ) without waiting for the end of each command? [duplicate]

This question already has an answer here:
Blocking and Non Blocking subprocess calls
(1 answer)
Closed 12 months ago.
There are two python scripts involved in this task.
My current task requires me to run a long process(takes about a day or two per each, and this is the first python script) in each of 29 available regions on GCP's instances. In order to finish the task as quick as possible, I'm trying to run each process in each instance all at once after spinning off 29 VMs all at once.
As manually running the first script by SSH-ing in to each of the instance is cumbersome, I wrote a python script(the second script) that SSHs into each region's VM and runs the first script I mentioned above.
The issue with the second script that runs first script in different regions is that it doesn't start off to run the first script in second region's VM until it finishes running in the first region's VM, whereas I need the second script to run the first script in every region without waiting for the process started by first script to end.
I use subprocess() in the second script to run the first script in each VMs.
The following code is the second script:
for zone, instance in zipped_zone_instance:
command = "gcloud compute ssh --zone " + zone + " " + instance + " --project cloud-000000 --command"
command_lst = command.split(" ")
command_lst.append("python3 /home/first_script.py")
subprocess.run(command_lst)
I need the subprocess.run(command_lst) to run for every 29 zones at once rather than it running for the second zone only after the first zone's process ends.
The following code is the first script:
for idx, bucket in enumerate(bucket_lst):
start = time.time()
sync_src = '/home/' + 'benchmark-' + var_
subprocess.run(['gsutil', '-m', '-o', 'GSUtil:parallel_composite_upload_threshold=40M', 'rsync', '-r', sync_src, bucket])
end = time.time() - start
time_lst.append(end)
tput_lst.append(tf_record_disk_usage / end)
What can I fix in the second script or the first script to achieve what I want??

Switch out your subprocess.run(command_lst) with Popen(command_lst, shell=True) in each of your scripts and and loop through the command list like the example below to run the processes in parallel.
This is how you implement Popen to run processes in parallel using arbitrary commands for simplicity.
from subprocess import Popen
commands = ['ls -l', 'date', 'which python']
processes = [Popen(cmd, shell=True) for cmd in commands]

Python: Process in one Thread stopping a Process in another Thread from finishing

I'd appreciate some help with threading, which I pretty new to.
The example code is not exactly what I’m doing (‘notepad’ and ‘calc’ are just example commands), but a simplified version that shows my problem.
I want to run two seperate threads that each run a different command a number of times. I would like the code to do this:
Start the first instance of ‘notepad’ and ‘calc’ simultaneously
(which it does)
When I close an instance of ‘notepad’, to open the
next instance of ‘notepad’.
When I close an instance of ‘calc’, to
open the next instance of ‘calc’.
[edit] I want the script to wait until both threads have finished, as it needs to do some processing of the output from these.
However, when I close an instance of ‘notepad’, the next instance of ‘notepad’ does not start until I’ve closed the current instance of ‘calc’ and vice versa. With a bit of de-bugging, it looks like the process (from Popen) for the closed instance of 'notepad' doesn't finish until the current 'calc' is closed.
Running Python 2.7 on Windows 7
Example Code:
from subprocess import Popen, PIPE, STDOUT
from threading import Thread
def do_commands(command_list):
for command in command_list:
proc = Popen("cmd.exe", stdin=PIPE, stdout=PIPE, stderr=STDOUT)
stdout_value, stderr_value = proc.communicate(input=command)
# MAIN CODE
A_command_list = ["notepad\n", "notepad\n", "notepad\n" ]
B_command_list = ["calc\n", "calc\n", "calc\n" ]
A_args = [A_command_list]
B_args = [B_command_list]
A_thread = Thread(target=do_commands, args=(A_args))
B_thread = Thread(target=do_commands, args=(B_args))
A_thread.start()
B_thread.start()
A_thread.join()
B_thread.join()
Thanks in advance :-)
Nick

So the communicate() method is apparently waiting for all processes created by Popen and executing cmd.exe and started at nearly the same time to terminate. Since the cmd.exe that runs calculator starts at nearly the same time as the cmd.exe that runs Notepad, both communicate() calls (one in A_thread and one in B_thread) wait until both processes term. Thus neither for loop advances until both processes term.
Adding a delay between starting the two threads fixes the problem.
So, leaving your original code unchanged and adding
sleep(1)
between the two Thread starts produces the desired behavior.
On my system, adding a delay of 0.0001 seconds reliably fixed the problem whereas a delay of 0.00001 did not.

Stop a running process in a Linux terminal through the Python subprocess module (airodump-ng) [duplicate]

This question already has answers here:
Using module 'subprocess' with timeout
(31 answers)
Closed 7 years ago.
I wrote a program (I ran it in the terminal) that goes through a list of terminal commands (Kali Linux).
import subprocess as sub
import time
sub.call(['airmon-ng', 'start', 'wlan0'])
p = sub.call(['airodump-ng','wlan0mon'])
time.sleep(10)
p.kill()
The last commmand is airodump-ng wlan0mon. Everything works fine (everything is displayed in the terminal (beacons, essid, etc.).
After a specified time I wish to kill the process (airodump-ng wlan0mon).
I don't want to press Ctrl + C by hand!
p.kill() does not work (maybe improper use).
How can I do this? What command should I send through the subprocess module?

The subprocess.call method is a high-level API that waits for the process to terminate before returning its exit code. If you need your main process to continue running while the subprocess runs, you need to use the slightly lower-level API: subprocess.Popen, which starts the process in the background.
Using p = sub.Popen(['airodump-ng','wlan0mon']) instead of p = sub.call(['airodump-ng','wlan0mon']) should work.

How to launch a couple of python scripts from a first python script and then terminate them all at once?

I have a function in a python script which should launch another python script multiple times, I am assuming this can be done like this(Script is just my imagination of how this would work.)
iterations = input("Enter the number of processes to run")
for x in range(0, iterations):
subprocess.call("python3 /path/to/the/script.py", shell=True)
but, I also need to pass over some defined variables into the other script, for example, if
x = 1
in the first script, then, I need x to have the same value in the second script without defining it there, I have NO idea how to do that.
And then also killing them, I have read about some method using PIDs, but don't those change every time?
Most of the methods I found on Google looked overly complex and what I want to do is really simple. Can anyone guide me in the right direction as to what to use and how I should go at accomplishing it?

I have a function in a python script which should launch another python script multiple times, I am assuming this can be done like this(Script is just my imagination of how this would work.)
**
Here is the subprocess manual page which contains everything I will be talking about
https://docs.python.org/2/library/subprocess.html
One of the way to call one script from other is using subprocess.Popen
something on the lines
import subprocess
for i in range(0,100):
ret = subprocess.Popen("python3 /path/to/the/script.py",stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
you can use the return value from Open to make the call synchronous using the communicate method.
out,err = ret.communicate()
This would block the calling script until the subprocess finishes.
I also need to pass over some defined variables into the other script??
There are multiple ways to do this.
1. Pass parameters to the called script and parse it using OptionPraser or sys.args
in the called script have something like
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-x","--variable",action="store_true",dest="xvalue",default=False)
(options,args) = parser.parse_args()
if options.xvalue == True:
###do something
in the callee script use subprocess as
ret = subprocess.Popen("python3 /path/to/the/script.py -x",stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
Note the addition of -x parameter
You can use args parse
https://docs.python.org/2/library/argparse.html#module-argparse
Pass the subprocess a environment variable which can be used to configure the subprocess. This is fast but this only works one way, i.e. from parent process to child process.
in called script
import os
x = int(os.enviorn('xvalue'))
in callee script set the environment variable
import os
int x = 1
os.environ['xvalue'] = str(x)
Use sockets or pipes or some other IPC method
And then also killing them, I have read about some method using PIDs, but don't those change every time?
again you can use subprocess to hold the process id and terminate it
this will give you the process id
ret.pid
you can then use .terminate to terminate the process if it is running
ret.terminate()
to check if the process is running you can use the poll method from subprocess Popen. I would suggest you to check before you terminate the process
ret.poll()
poll will return a None if the process is running

If you just need to pass some values to second script, and you need to run that
by means of subprocess module, then you may simply pass the variables as command line arguments:
for x in range(0, iterations):
subprocess.call('python3 /path/to/second_script.py -x=%s'%x, shell=True)
And recieve the -x=1 via sys.argv list inside second_script.py (using argparse module)
On the other hand, If you need to exchange something between the two scripts dynamically (while both are running), You can use the pipe mechanism or even better, use the multiprocessing (wich requires some changes in your current code), it would make communication with and controlling it (terminating it) much cleaner.

You can pass variables to subprocesses via the command line, environment variables or passing data in on stdin. Command line is easy for simple strings that aren't too long and don't themselves have shell meta characters in them. The target script would pull them from sys.argv.
script.py:
import sys
import os
import time
x = sys.argv[1]
print(os.getpid(), "processing", x)
time.sleep(240)
subprocess.Popen starts child processes but doesn't wait for them to complete. You could start all of the children, put their popen objects in a list and finish with them later.
iterations = input("Enter the number of processes to run")
processes = []
for x in range(0, iterations):
processes.append(subprocess.Popen([sys.executable, "/path/to/the/script.py", str(x)])
time.sleep(10)
for proc in processes:
if proc.poll() is not None:
proc.terminate()
for proc in processes:
returncode = proc.wait()

Running a command completely indepenently from script [duplicate]

This question already has answers here:
Run a program from python, and have it continue to run after the script is killed
(5 answers)
Closed 8 years ago.
I have recently come across some situations where I want to start a command completely independently, and in a different process then the script, equivalent typing it into a terminal, or more specifically writing the command into a .sh or .desktop and double clicking it. The request I have are:
I can close the python window without closing the application.
The python window does not display the stdout from the application (you don't want random text appearing in a CLI), nor do I need it!.
Things I have tried:
os.system waits.
subprocess.call waits.
subprocess.Popen starts a subprocess (duh) closing the parent process thus quits the application
And thats basically all you can find on the web!
if it really comes down to it I could launch a .sh (or .bat for windows), but that feels like a cheap hack and not the way to do this.

If you were to place an & after the command when called from os.system, would that not work? For example:
import os
os.system( "google-chrome & disown " )

import subprocess
import os
with open(os.devnull, 'w') as f:
proc = subprocess.Popen(command, shell=True, stdout=f, stderr=f)

Spawn a child process "the UNIX way" with fork and exec. Here's a simple example of a shell in Python.
import os
prompt = '$ '
while True:
cmds = input(prompt).split()
if len(cmds):
if (os.fork() == 0):
# Child process
os.execv(cmds[0], cmds)
else:
# Parent process
# (Hint: You don't have to wait, you can just exit here.)
os.wait()