I am writing a Python tool that will automatically do a number of operations for me. One part of the automation is the parallel processing of n number of LSDYNA finite element simulations. I want to send the n simulations to a pool and have them distributed to a user specified number of processors, x. As one simulation terminates, another should be sent from the pool to the idle processor until the pool is empty. At this time, the rest of the python code should continue to execute. If this code were working correctly on my Windows machine, I would expect to see x number of cmd windows running LSDYNA at any time (until to pool is empty).
I have read a number of similar questions and they all seem to end up using the multiprocessing module. I've written a code that I thought was correct however, when I execute it, nothing happens. There are no error messages in the terminal window, and I do not get any LSDYNA output.
I do have a windows batch script that does work and does the same thing, in case that would be helpful to anyone.
In case I'm doing something completely wrong, a note about LSDYNA: When run via command line, each simulation runs in its own terminal window. The output files are written to the current directory at the time of command execution. The command format is:
"C:\LSDYNA\program\ls971_s_R5.1.1_winx64_p.exe" i=input.k ncpu=1 memory=100m
This is the Python code that I've come up with:
import os
import multiprocessing as mp
import subprocess
import glob
def run_LSDYNA(individual_sim_dir, model_name, solver_path):
os.chdir(individual_sim_dir)
os.environ['lstc_license'] = 'network'
os.environ['lstc_license_server'] = 'xxx.xx.xx.xx'
subprocess.call([solver_path, "i=%s" % model_name, "ncpu=1", "memory=100m"])
def parallel(sim_dir, procs, model_name, solver_path):
run_dirs = []
for individual_sim_dir in glob.glob(os.path.join(sim_dir, 'channel_*')):
run_dirs.append(individual_sim_dir)
pool = mp.Pool(processes=procs)
args = ((run_dir, model_name, solver_path) for run_dir in run_dirs)
simulations = pool.map_async(run_LSDYNA, args)
simulations.wait()
if __name__ == "__main__":
mp.freeze_support()
parallel('C:\Users\me\Desktop\script_test\Lower_Leg_Sims', 2, 'boot.k', "C:\LSDYNA\program\ls971_s_R5.1.1_winx64_p.exe")
Related
I am attempting to write a python function that uses multiprocessing to execute multiple instances of a external serial software. Using the code below I am able to "start" all of the runs of the external software, but it appears that only 1 instance of the software is actually computing (i.e., log files were generated for every run but only one instance is updating the log). One issue with the external program that I am using is that it only accepts one specific input file name. My original solution to this is to start the Processes 1 by 1 while waiting 10 seconds in between, so that I can update the information of this input file after each process instance has started. In otherwords, I write a new input file each instance starts.
Overall, what I would like to do is allocate X # of CPUs (via a scheduler) to my python program and then have my python program start X # of serial instances of the external software, so that all instances run in parallel. I should also not return to the main python program until all instances of the external software have completed. I appreciate any helpful insights! I am relatively new to this type of parallelism, so I apologize if my question has an exceptionally simple solution that I don't see.
from multiprocessing import Process
from subprocess import call, Popen, PIPE
import time
def run_parallel_serial(n_loops=None, name=None, setting=None):
processes = []
for i in range(1, n_loops+1, 1):
prepare_software_input(n_loop=i, info=name)
processes.append(Process(target=run_software))
count = 0
for p in processes:
update_input(settings[count])
p.start()
time.sleep(10)
count += 1
for p in processes:
p.join()
print("Done running in parallel!")
return
The uprepare_software_input and update_input functions are used to write and overwrite the input file used by the external software. The run_software function uses the subprocess module as the following
def run_software():
p = Popen([PROGRAM_EXEC, SOFTWARE_INPUT_FILE], stdin=PIPE, stdout=PIPE, stderr=PIPE).communicate()
del(p)
return
p.s. I am not married to multiprocessing. If there is an alternative strategy with a different module, I am happy to switch.
I have a website (Wordpress site on Ubuntu OS and Apache Server) with special math calculators, many of which utilize python3 scripts to do the main calculations. The flow of data on these calculators is as such:
1.) User inputs numbers into html form, then hits submit button.
2.) PHP function is called, it assigns html user inputs to variables and does exec() on applicable python3 file with those variables (the user inputs are filtered and escapeshellarg is used so all good here).
3.) PHP function returns result of python3 script which is displayed via shortcode on the calculator web page.
The issue I am having is that occasionally the symbolic and numeric computations within my python scripts will hang up indefinitely. As that python3 process keeps running, it starts to use massive CPU and memory resources (big problem during peak traffic hours).
My question is this: is there some way to make a script or program on my server's backend that will kill a process instance of python3 if it has exceeded an arbitrary runtime and CPU usage level? I would like to restrict it only to instances of python3 so that it can't kill something like mysqld. Also, I am OK if it only uses runtime as a kill condition. None of my python scripts should run longer than ~10 seconds under normal circumstances and CPU usage will not be an issue if they don't run longer than 10 seconds.
You can create another python script to serve as a health checker on your server based on the psutil and os modules.
The following code could serve as a base for your specific needs, notice that what it does is basically check for the PIDs for the python scripts on the script_name_list variable based on the name of the script and kill them after checking if your server's CPU is above some threshold or if the memory available is below some threshold as well.
#!/usr/bin/env python3
import psutil
import os
import signal
CPU_LIMIT = 80 #Change Me
AV_MEM = 500.0 #Change Me
script_name_list = ['script1'] #Put in the name of the scripts
def find_other_scripts_pid(script_list):
pid_list = []
for proc in psutil.process_iter(['pid','name', 'cmdline']):
#this is not the PID of the process referencing this script and therefore we chould check inside the list of script name to kill them
if proc.info['pid'] != os.getpid() and proc.info['name'] in ['python','python3']:
for element in proc.info['cmdline']:
for script_name in script_name_list:
if script_name in element:
pid_list.append(proc.info['pid'])
return pid_list
def kill_process(pid):
if psutil.pid_exists(pid):
os.kill(pid,signal.SIGKILL)
return None
def check_cpu():
return psutil.cpu_percent(interval=1)
def check_available_memory():
mem = psutil.virtual_memory()
return mem.available/(2**(20))
def main():
cpu_usage = check_cpu()
av_memory_mb = check_available_memory()
if cpu_usage > CPU_LIMIT or av_memory_mb < AV_MEM:
pid_list = find_other_scripts_pid(script_name_list)
for pid in pid_list:
kill_process(pid)
if __name__ == "__main__":
main()
You can afterwards run this script periodically on your server by using a crontab as explained on this post shared within the community.
I'm trying to run multiple exe's (12 of them), because of computer resources I can spawn maximum 4 at a time before I get performance degradation.
I'm trying to find if there is a way to call 4 exe's at a time and as soon as one of them finishes, to call another exe to fill the resources that have freed up
My current code does this:
excs = [r"path\to\exe\exe.exe",r"path\to\exe\exe.exe",r"path\to\exe\exe.exe",r"path\to\exe\exe.exe"]
running = [subprocess.Popen(ex) for ex in excs]
[process.wait() for process in running]
It repeats this process three times so that it runs all 12. Unfortunately it means that it needs to wait for all of them to finish before moving on to the next set. Is there a more efficient way of doing this?
For the record, all of the exe's have different run times.
Python has ThreadPoolExecutor which makes this very convenient
import subprocess
from concurrent.futures import ThreadPoolExecutor
def create_pool(N,commands):
pool = ThreadPoolExecutor(max_workers=N)
for command in commands:
pool.submit(subprocess.call, command)
pool.shutdown(wait=False)
def main():
N_WORKERS=4
commands = [job1, job2, ...]
create_pool(N_WORKERS, commands)
I am trying to use the python multiprocessing library in order to parallize a task I am working on:
import multiprocessing as MP
def myFunction((x,y,z)):
...create a sqlite3 database specific to x,y,z
...write to the database (one DB per process)
y = 'somestring'
z = <large read-only global dictionary to be shared>
jobs = []
for x in X:
jobs.append((x,y,z,))
pool = MP.Pool(processes=16)
pool.map(myFunction,jobs)
pool.close()
pool.join()
Sixteen processes are started as seen in htop, however no errors are returned, no files written, no CPU is used.
Could it happen that there is an error in myFunction that is not reported to STDOUT and blocks execution?
Perhaps it is relevant that the python script is called from a bash script running in background.
The lesson learned here was to follow the strategy suggested in one of the comments and use multiprocessing.dummy until everything works.
At least in my case, errors were not visible otherwise and the processes were still running as if nothing had happened.
I have a binary (say a.out) that I want to call with different configs. I want to run these configs on a 40-core machine in parallel. Below is a sketch of my code.
It is very straightforward: I generate a config and pass in into the worker, and the worker calls the binary with the config using subprocess. I am also redirecting the output to a file. Let's call this piece of code run.py
def worker(cmdlist, filename):
outputfile = open(filename, 'wb')
// here it essentially executes a.out config > outputfile
subprocess.call(cmdlist, stderr=outputfile, stdout=outputfile)
outputfile.close()
def main():
pool = Pool(processes = 40)
for config in all_configs
filename, cmdlist = genCmd(config)
res = pool.apply_async(worker, [cmdlist, filename])
results.append(res)
for res in results:
res.get()
pool.close()
But after I kick it off, I realized that I am not spawning as many processes as I want. I definitely submitted more than 40 workers, but in top, I am only seeing about 20 of a.out.
I do see many of the run.py that are in "sleeping" state (i.e., "S" in top). When I do a ps auf, I also saw a lot of run.py in "S+" state, with no binary spawned out. Only about half of them spawned "a.out"
I am wondering, why is this happening? I am redirecting the output to a network-mounted hard-drive, which could be a reason, but in top I only see 10%wa (which in my understanding is 10% of the time waiting for IO). I don't think this results in 50% of idle CPUs. Plus, I should at least have the binary spawned out, instead of being stuck at run.py. My binary's runtime is also long enough. I should really be seeing 40 jobs running for a long time.
Any other explanation? Anything I did wrong in my python code?
An approach I have used to make use of many simultaneous processes running at once on multiple cores is to use p = subprocess.Popen(...) and p.Poll(). In your case I think you would be able to skip using Pool altogether. I'd give you a better example but unfortunately I don't have access to that code anymore.