I'm on Windows environment, and suppose I have two toy programs, called 2.bat and 5.bat, which look like timeout 2 and timeout 5, respectively.
I want to set up a script that runs both 2.bat and 5.bat in parallel, and when 2.bat finishes it is ran again and likewise for 5.bat. I'm pretty bad at Python, but after a bit of searching I see that I can do:
from subprocess import Popen
commands = ["2.bat", "5.bat"]
while True:
procs = [Popen(i) for i in commands]
for p in procs:
p.wait()
This doesn't do what I want: it waits for both processes to finish, and then again executes both. What I want to do (in pseudocode) is as follows:
while True:
in parallel, run 2.bat and 5.bat
when 2.bat finishes, rerun 2.bat again
when 5.bat finishes, rerun 5.bat again
Can I achieve this with subprocess, or do I need other libraries?
my solution would be:
from _thread import start_new_thread
from subprocess import Popen
commands = ["2.bat", "5.bat"]
def run_bat(file):
while True:
p = Popen(file)
p.wait()
for command in commands:
start_new_thread(run_bat, (command, ))
while True:
pass
Related
I have two scripts Server.py and ServerGUI.py. I want them to run independently and in parallel. Say I make another script main.py. How can I run Server.py and ServerGUI.py from main.py?
Can you suggest me the code for main.py?
To run 2 or more scripts from within a python script, you can use the subprocess package with nohup. This will run each script in the background allowing you to run them in parallel from the same source script. Also, as an option, this example will save the standard output from each script in a different file
import os
from subprocess import call
from subprocess import Popen
# subprocess.call(['python', 'exampleScripts.py', somescript_arg1, somescript_val1,...]).
Popen(['nohup', 'python', 'exampleScripts.py'],
stdout=open('null1', 'w'),
stderr=open('logfile.log', 'a'),
start_new_session=True )
Popen(['nohup', 'python', 'exampleScripts.py'],
stdout=open('null2', 'w'),
stderr=open('logfile.log', 'a'),
start_new_session=True )
Popen(['nohup', 'python', 'exampleScripts.py'],
stdout=open('null3', 'w'),
stderr=open('logfile.log', 'a'),
start_new_session=True )
Output: the start and end times in each script overlap showing that the 2nd one started before the first one ended
(ds_tensorflow) C:\DataScience\SampleNotebooks\Threading>python RunScripts.py
(ds_tensorflow) C:\DataScience\SampleNotebooks\Threading>cat null*
2020-07-13 15:46:21.251606
List processing complete.
2020-07-13 15:46:29.130219
2020-07-13 15:46:22.501599
List processing complete.
2020-07-13 15:46:31.227954
2020-07-13 15:46:23.758498
List processing complete.
2020-07-13 15:46:32.431079
You can also use the same idea with functions, if you want to keep the code in once place. This example will run two different functions two times, in parallel.
Function example:
...
import threading
...
def some_function()
# code
def other_function()
# code
if __name__ == "__main__":
jobs = []
#same function run multiple times
threads = 2
for i in range(0, threads):
out_list = list()
thread1 = threading.Thread(target=some_function(size, i, out_list))
jobs.append(thread1)
thread2 = threading.Thread(target=other_function(size, i, out_list))
jobs.append(thread2)
# Start the threads (i.e. calculate the random number lists)
for j in jobs:
j.start()
# Ensure all of the threads have finished
for j in jobs:
j.join()
# continue processing
You can use threading or multiprocessing 'for running the python scripts in parallel.
Threading : https://www.tutorialspoint.com/python/python_multithreading.htm
Multiprocessing :
https://www.tutorialspoint.com/multiprocessing-in-python#:~:text=The%20multiprocessing%20package%20supports%20spawning,is%20similar%20to%20threading%20module.
Hope this helps you
You could do something along these lines (for the sake of example, I have assumed your scripts accept two arguments, arg1 and arg2. This needs to be modified according to your specific needs.):
If you have "main" functions in server.py and servergui.py:
import threading
from server import server_main
from servergui import server_gui_main
thread_list = []
thread_list.append(
threading.Thread(target=server_main, args=(arg1, arg2))
)
thread_list.append(
threading.Thread(target=server_gui_main, args=(arg1, arg2))
)
for thread in thread_list:
thread.start()
for thread in thread_list:
thread.join()
The above will start the two main functions in separate threads.
If you want separate parallel processes use multithreading:
import multiprocessing
process_list = []
process_list.append(
multiprocessing.Process(target=server_main, args=(arg1, arg2))
)
process_list.append(
multiprocessing.Process(target=server_gui_main, args=(arg1, arg2))
)
for process in process_list:
process.start()
for process in process_list:
process.join()
As you see, the differences in the API of multiprocessing and threading are small. However, your performance may suffer from threading if you are performing CPU-bound tasks. This is because the Python GIL forces Python to run only one thread at any given moment. Thus, if you have CPU-intensive tasks you should use multiprocessing as this creates separate processes which do indeed run in parallel.
If you want to start server.py and servergui.py as if you start them from the command line:
import subprocess
subprocess.run(
['python', 'server.py', 'arg1', 'arg2'],
shell=True
)
subprocess.run(
['python', 'servergui.py', 'arg1', 'arg2'],
shell=True
)
In below code from a Python script I'm spawning another Python script.
I want to run "test_bin" in the background and don't want to call wait or communicate to reap the exit status. If I not used wait/communicate "test_bin" become defunct(zombie) after completion.
Is there any way we can avoid defunct without wait/communicate?
import subprocess
test_bin = "/usr/local/bin/test_code.py" # Took 5-10 min to complete
proc_handle = subprocess.Popen( (test_bin, shell=True)
:
:
proc_handle.wait() # don't want to do this.
In Unix-like systems, it is enough to ignore SIGCHLD for child processes to never become zombies. So you could do:
import subprocess
import signal
test_bin = "/usr/local/bin/test_code.py" # Took 5-10 min to complete
signal.signal(signal.SIGCHLD, signal.SIG_IGN)
proc_handle = subprocess.Popen( (test_bin, shell=True)
...
I am new to python and using multiprocessing, I am starting one process and calling one shell script through this process. After terminating this process shell script keeps running in the background, how do I kill it, please help.
python script(test.py)
#!/usr/bin/python
import time
import os
import sys
import multiprocessing
# test process
def test_py_process():
os.system("./test.sh")
return
p=multiprocessing.Process(target=test_py_process)
p.start()
print 'STARTED:', p, p.is_alive()
time.sleep(10)
p.terminate()
print 'TERMINATED:', p, p.is_alive()
shell script (test.sh)
#!/bin/bash
for i in {1..100}
do
sleep 1
echo "Welcome $i times"
done
The reason is that the child process that is spawned by the os.system call spawns a child process itself. As explained in the multiprocessing docs descendant processes of the process will not be terminated – they will simply become orphaned. So. p.terminate() kills the process you created, but the OS process (/bin/bash ./test.sh) simply gets assigned to the system's scheduler process and continues executing.
You could use subprocess.Popen instead:
import time
from subprocess import Popen
if __name__ == '__main__':
p = Popen("./test.sh")
print 'STARTED:', p, p.poll()
time.sleep(10)
p.kill()
print 'TERMINATED:', p, p.poll()
Edit: #Florian Brucker beat me to it. He deserves the credit for answering the question first. Still keeping this answer for the alternate approach using subprocess, which is recommended over os.system() in the documentation for os.system() itself.
os.system runs the given command in a separate process. Therefore, you have three processes:
The main process in which your script runs
The process in which test_py_processes runs
The process in which the bash script runs
Process 2 is a child process of process 1, and process 3 is a child of process 1.
When you call Process.terminate from within process 1 this will send the SIGTERM signal to process two. That process will then terminate. However, the SIGTERM signal is not automatically propagated to the child processes of process 2! This means that process 3 is not notified when process 2 exits and hence keeps on running as a child of the init process.
The best way to terminate process 3 depends on your actual problem setting, see this SO thread for some suggestions.
I'm working with a Python script and I have some problems on delaying the execution of a Bash script.
My script.py lets the user choose a script.sh, and after the possibility to modify that, the user can run it with various options.
One of this option is the possibility to delay of N seconds the execution of the script, I used time.sleep(N) but the script.py totally stops for N seconds, I just want to retard the script.sh of N seconds, letting the user continue using the script.py.
I searched for answers without success, any ideas?
You can start the script in a New thread, sleeping before running it.
Minimal example:
import subprocess as sp
from threading import Thread
import time
def start_delayed(args, delay):
time.sleep(delay)
sp.run(args)
t = Thread(target=start_delayed, kwargs={'args': ['ls'], 'delay': 5})
t.start()
Consider using a Timer object from the threading module:
import subprocess, threading
t = threading.Timer(10.0, subprocess.call, args=(['script.sh'],))
t.start()
...the above running script.sh after a 10-second delay.
Alternately, if you want to efficiently be able to run an arbitrary number of scheduled tasks with only a single thread controlling them, consider using a scheduler from the tandard-library sched module:
import sched, subprocess
s = sched.scheduler(time.time, time.sleep)
s.enter(10, subprocess.call, (['script.sh'],))
s.run()
This will run script.sh after 10 seconds have passed -- though if you want it to run in the background, you'll want to put it in a thread (or such) yourself.
You should run sleep using subprocess.Popen before calling script.sh.
I am trying to migrate a bash script to Python.
The bash script runs multiple OS commands in parallel then waits for them to finish before resuming, ie:
command1 &
command2 &
.
commandn &
wait
command
I want to achieve the same using Python subprocess. Is this possible? How can I wait for a subprocess.call command to finish before resuming?
You can still use Popen which takes the same input parameters as subprocess.call but is more flexible.
subprocess.call: The full function signature is the same as that of the Popen constructor - this functions passes all supplied arguments directly through to that interface.
One difference is that subprocess.call blocks and waits for the subprocess to complete (it is built on top of Popen), whereas Popen doesn't block and consequently allows you to launch other processes in parallel.
Try the following:
from subprocess import Popen
commands = ['command1', 'command2']
procs = [ Popen(i) for i in commands ]
for p in procs:
p.wait()
Expanding on Aaron and Martin's answer, here is a solution that runs uses subprocess and Popen to run n processes in parallel:
import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5']
n = 2 #the number of parallel processes you want
for j in range(max(int(len(commands)/n), 1)):
procs = [subprocess.Popen(i, shell=True) for i in commands[j*n: min((j+1)*n, len(commands))] ]
for p in procs:
p.wait()
I find this to be useful when using a tool like multiprocessing could cause undesired behavior.