I'm new on python. I want to learn how to parallel processing in python. I saw the following example:
import multiprocessing as mp
np.random.RandomState(100)
arr = np.random.randint(0, 10, size=[20, 5])
data = arr.tolist()
def howmany_within_range_rowonly(row, minimum=4, maximum=8):
count = 0
for n in row:
if minimum <= n <= maximum:
count = count + 1
return count
pool = mp.Pool(mp.cpu_count())
results = pool.map(howmany_within_range_rowonly, [row for row in data])
pool.close()
print(results[:10])
but when I run it, this error happened:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
What should I do?
If you place everything in global scope inside this if __name__ == "__main__" block as follows, you should find that your program behaves as you expect:
def howmany_within_range_rowonly(row, minimum=4, maximum=8):
count = 0
for n in row:
if minimum <= n <= maximum:
count = count + 1
return count
if __name__ == "__main__":
np.random.RandomState(100)
arr = np.random.randint(0, 10, size=[20, 5])
data = arr.tolist()
pool = mp.Pool(mp.cpu_count())
results = pool.map(howmany_within_range_rowonly, [row for row in data])
pool.close()
print(results[:10])
Without this protection, if your current module was imported from a different module, your multiprocessing code would be executed. This could occur within a non-main process spawned in another Pool and spawning processes from sub-processes is not allowed, hence we protect against this problem.
I had a live example, where I faced the same RuntimeError issue when I executed a specific tool on MacOS-machines (on Linux machines it was fine though). However, I'm not sure about the exact cause for the problem, cause the if __name__ == "__main__" encapsulation seemed to be properly at place.
Following one comment on this Stack-Overflow entry, I suspected that using python>=3.8, which utilizes spawn as default method for calling subprocesses might be the problem.
My solution:
Using python=3.7 did the trick.
Related
I am multiprocessing a for loop in Python using Pool in this way:
if __name__ == '__main__':
pool = Pool() # Create a multiprocessing Pool
result = pool.map(process_interp, range(ngal))
where process_interp is a function I have defined. However, when the number of calculations increase, the program seems to almost crash in a way that a 0 byte file is produced as my result and the Python program continues running but htop shows that it is no longer using multiple threads and only a single thread is being used. For not so heavy calculations, everything just works fine and the program runs fast and correctly.
Can someone please give me insight into what is happening here and what causes this problem? I have no clue how to solve this issue. Thanks a lot!
Edit: This is the function process_interp. Basically calculating some values on a grid. The issue arises when I increase the size of the grid (N_a and N_m) and therefore the number of calculations.
def process_interp(k):
#e = data[k][0]+np.random.normal(0, 0.3, len(data[k][0]))
e = data[k][0]
sigma_crit = data[k][1]
r_s = data[k][2]
dev_gamma = data[k][4]
dev_k = data[k][5]
z_d = data[k][6]
like_grid = np.empty((N_a, N_m), dtype='longdouble')
for i in range(N_a):
for j in range(N_m):
gamma_nfw, k_nfw = NFWfunc(m_grid[j], z_d, r_s, sigma_crit)
gamma_dev = dev_gamma*pow(10, alpha_grid[i]-alpha_0)
k_dev = dev_k*pow(10, alpha_grid[i]-alpha_0)
mean = (gamma_nfw+gamma_dev)/(1-k_nfw-k_dev)
loglike = np.zeros(len(e), dtype='longdouble')
loglike = -((mean-e)**2/(2*0.3**2))
like_grid[i,j] = np.prod(np.exp(loglike))
return like_grid
Is there a way to run a function in parallel within an already parallelised function? I know that using multiprocessing.Pool() this is not possible as a daemonic process can not create a child process. I am fairly new to parallel computing and am struggling to find a workaround.
I currently have several thousand calculations that need to be run in parallel using some other commercially available quantum mechanical code I interface to. Each calculation, has three subsequent calculations that need to be executed in parallel on normal termination of the parent calculation, if the parent calculation does not terminate normally, that is the end of the calculation for that point. I could always combine these three subsequent calculations into one big calculation and run normally - although I would much prefer to run separately in parallel.
Main currently looks like this, run() is the parent calculation that is first run in parallel for a series of points, and par_nacmes() is the function that I want to run in parallel for three child calculations following normal termination of the parent.
def par_nacmes(nacme_input_data):
nacme_dir, nacme_input, index = nacme_input_data # Unpack info in tuple for the calculation
axes_index = get_axis_index(nacme_input)
[norm_term, nacme_outf] = util.run_calculation(molpro_keys, pwd, nacme_dir, nacme_input, index) # Submit child calculation
if norm_term:
data.extract_nacme(nacme_outf, molpro_keys['nacme_regex'], index, axes_index)
else:
with open('output.log', 'w+') as f:
f.write('NACME Crashed for GP%s - axis %s' % (index, axes_index))
def run(grid_point):
index, geom = grid_point
if inputs['code'] == 'molpro':
[spe_dir, spe_input] = molpro.setup_spe(inputs, geom, pwd, index)
[norm_term, spe_outf] = util.run_calculation(molpro_keys, pwd, spe_dir, spe_input, index) # Run each parent calculation
if norm_term: # If parent calculation terminates normally - Extract data and continue with subsequent calculations for each point
data.extract_energies(spe_dir+spe_outf, inputs['spe'], molpro_keys['energy_regex'],
molpro_keys['cas_prog'], index)
if inputs['nacme'] == 'yes':
[nacme_dir, nacmes_inputs] = molpro.setup_nacme(inputs, geom, spe_dir, index)
nacmes_data = [(nacme_dir, nacme_inp, index) for nacme_inp in nacmes_inputs] # List of three tuples - each with three elements. Each tuple describes a child calculation to be run in parallel
nacme_pool = multiprocessing.Pool()
nacme_pool.map(par_nacmes, [nacme_input for nacme_input in nacmes_data]) # Run each calculation in list of tuples in parallel
if inputs['grad'] == 'yes':
pass
else:
with open('output.log', 'w+') as f:
f.write('SPE crashed for GP%s' % index)
elif inputs['code'] == 'molcas': # TO DO
pass
if __name__ == "__main__":
try:
pwd = os.getcwd() # parent dir
f = open(inp_geom, 'r')
ref_geom = np.genfromtxt(f, skip_header=2, usecols=(1, 2, 3), encoding=None)
f.close()
geom_list = coordinate_generator(ref_geom) # Generate nuclear coordinates
if inputs['code'] == 'molpro':
couplings = molpro.coupled_states(inputs['states'][-1])
elif inputs['code'] == 'molcas':
pass
data = setup.global_data(ref_geom, inputs['states'][-1], couplings, len(geom_list))
run_pool = multiprocessing.Pool()
run_pool.map(run, [(k, v) for k, v in enumerate(geom_list)]) # Run each parent calculation for each set of coordinates
except StopIteration:
print('Please ensure goemetry file is correct.')
Any insight on how to run these child calculations in parallel for each point would be a great help. I have seen some people suggest using multi-threading instead or to set daemon to false, although I am unsure if this is the best way to do this.
firstly I dont know why you have to run par_nacmes in paralel but if you have to you could:
a use threads to run them instead of processes
or b use multiprocessing.Process to run run however that would involve a lot of overhead so I personally wouldn't do it.
for a all you have to do is
replace
nacme_pool = multiprocessing.Pool()
nacme_pool.map(par_nacmes, [nacme_input for nacme_input in nacmes_data])
in run()
with
threads = []
for nacme_input in nacmes_data:
t = Thread(target=par_nacmes, args=(nacme_input,)); t.start()
threads.append(t)
for t in threads: t.join()
or if you dont care if the treads have finished or not
for nacme_input in nacmes_data:
t = Thread(target=par_nacmes, args=(nacme_input,)); t.start()
I am running a multiprocessing pool in a for loop over a chuck of data. It runs fine for two iterations and hangs on the third. If I reduce the size of each chuck it hangs later on perhaps the forth or fifth iteration. In the program where I discovered the problem I am running a more extensive function but this works to reproduce the error.
Is there a proper way to terminate a pool after it is finished? So that I can start it again.
import pandas as pd
import numpy as np
from multiprocess import Pool
df = pd.read_csv('paths.csv')
def do_something(user):
v = df[df['userId'] == user]
return v
if __name__ == '__main__':
users = df['userId'].unique()
n_chunks = round(len(users)/40)
subsets = [users[i:i+n_chunks] for i in range(0, len(users), n_chunks)]
chunk_counter = 0
for user_subset in subsets:
chunk_counter += 1
print(f'Beginning to process chunk {chunk_counter}...')
with Pool() as pool:
frames = pool.map(do_something, user_subset)
pool.close()
pool.terminate()
print(f'Completed processing chunk {chunk_counter}.')
I was able to prevent the hanging with the code below:
with Pool(maxtasksperchild=1) as pool:
frames = pool.map_async(do_something, user_subset).get()
pool.terminate()
pool.join()
I don't understand why using map_async would prevent the hanging. I will dive deeper if I have a chance and update if I understand the reason.
I wrote some code to break up a for loop into multiple processes to speed up calculations.
import numpy as np
import formfactors
from subdivide_loop import subdivide_loop
import multiprocessing
def worker(start, end, triangleI, areaI, scene, kdtree, samples, output):
form_factors = np.zeros(end-start)
for j in range(start, end):
triangleJ = np.array(scene[j][0:4])
form_factors[start] = formfactors.uniform(triangleJ, triangleI, areaI, kdtree, samples)
result = output.get(block=True)
for j in range(start, end):
result[j] = form_factors[j]
output.put(result)
def calculate_formfactors(start, end, triangleI, areaI, scene, kdtree, samples, output, nb_processes,
max_interval_length):
intervals = subdivide_loop(start, end, max_interval_length, nb_processes)
print("start")
jobs = []
for k in range(nb_processes):
p = multiprocessing.Process(target=worker,
args=(intervals[k][0], intervals[k][1], triangleI, areaI, scene, kdtree,
samples, output))
jobs.append(p)
for p in jobs:
p.start()
for p in jobs:
p.join()
results = output.get()
return results
I would like to be able to call calculate_formfactors() inside a function inside a loop, like this:
def outer_function():
for i in range(1000):
for j in range(i + 1, 1000, max_interval_length):
form_factors = calculate_formfactors(args)
But running this gives an error:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
Because of how the outer function works, breaking up outer_function() instead of calculate_formfactors() is not possible.
So, any advice on how to do this?
As the error suggests, make sure your outer_function() (or whatever initiates it) is called from within an __main__ guard, e.g.
if __name__ == "__main__":
outer_function()
It doesn't have to be the outer_function() but you need to trace it all back to the first step that initializes the chain that ultimately leads to the call to multiprocessing.Process() and put it within the above block.
This is because on non-forking systems multiple processes are essentially run as subprocesses so creating new processes from the main script would end up with an infinite recursion/processes spawning. You can read more about it in this answer. Because of that, you have to make sure your multiprocessing initialization code executes only once which is where the __main__ guard comes in.
I have a for loop, which uses some binary conditions and finally writes a file accordingly. The problem I have is, the conditions are true for many files (sometimes around 1000 files need to be written). So writing them takes a long time (around 10 mins). I know I can somehow use Python's multiprocessing and utilise some of the cores.
This is the code that works, but only uses one core.
for i,n in enumerate(halo_param.strip()):
mask = var1['halo_id'] == n
newtbdata = tbdata1[mask]
hdu = pyfits.BinTableHDU(newtbdata)
hdu.writeto(('/home/Documments/file_{0}.fits').format(i))
I came across that it can be done using Pool from multiprocessing.
if __name__ == '__main__':
pool = Pool(processes=4)
I would like to know how to do it and utilise atleast 4 of my cores.
Restructure the for loop body as a function, and use Pool.map with the function.
def work(arg):
i, n = arg
mask = var1['halo_id'] == n
newtbdata = tbdata1[mask]
hdu = pyfits.BinTableHDU(newtbdata)
hdu.writeto(('/home/Documments/file_{0}.fits').format(i))
if __name__ == '__main__':
pool = Pool(processes=4)
pool.map(work, enumerate(halo_param.strip()))
pool.close()
pool.join()