Develop Reference

Parallelizing a function to write to global variables

I'm trying to figure out the best way to parallelize a program like this:
global_data = some data
global_data2 = some data
data_store1 = np.empty(n)
data_store2 = np.empty(n)
.
.
.
def simulation(global_data):
retrieve values from global datasets and set element of global datastores
such that I do something like pass list(enumerate(global_data)) to a multiprocessing function, and each process sets elements of the global data stores corresponding to the received (index, vlaue) pair. I'm running on a high performance cluster with 128 cores, so I think parallelization is preferable to threading.

If you use a multiprocessing pool (e.g. a multiprocessing.Pool instance) with its map method, then your worker function, simulation, just needs to return its result back to the main process which will end up with a list of results that will be in the correct order. This will be less expensive than using a managed list to which the worker function adds its result:
import multiprocessing
def simulation(global_data_elem):
# We are passed a single element of global_data
# Do calculation with global_data_elem and return result.
# The CPU resources required to do this calculation must be sufficiently
# high to justify the additional overhead of multiprocessing (which is
# not the case for this demo):
return global_data_elem * 2
def main():
# global_data is some data (not necesarilly at global scope)
global_data = ['a', 'b', 'c']
# create pool of the correct size
# (not larger than the number of cores we have nor the number of tasks being submitted):
pool = multiprocessing.Pool(min(len(global_data), multiprocessing.cpu_count()))
# results are returned in the correct order (task submission order):
results = pool.map(simulation, global_data)
print(results)
# Required for Windows:
if __name__ == '__main__':
main()
Prints:
['aa', 'bb', 'cc']

QTimer.singleshot is causing the GUI to hangout

I am creating a table that shows the running processes, with a decorator to update these information. Using the decorator as in the code below, causes the GUI to hangs out every time the singleshot is called (every second).
Why the singleshot is causing the GUI to hangs, and how can I get better logic ?
# First create table
data = getProcesses()
tableWidget = QTableWidget()
Layout.addWidget(tableWidget)
fillTable(data, len(data['pid']), len(data), tableWidget)
# get the processes
def getProcesses():
allprocesses = {}
for p in psutil.process_iter():
try:
if p.name().lower() in ["python.exe", "pythonw.exe"]: # console, window
with p.oneshot():
allprocesses.setdefault('pid', []).append(p.pid)
allprocesses.setdefault('memory(MB)', []).append(p.memory_full_info().uss/(1024**2))
allprocesses.setdefault('memory(%)', []).append(p.memory_percent(memtype="rss"))
allprocesses.setdefault('cpu_times(s)', []).append(sum(p.cpu_times()[:2]))
allprocesses.setdefault('create_time', []).append(datetime.datetime.fromtimestamp(p.create_time()).strftime("%Y-%m-%d %H:%M:%S"))
allprocesses.setdefault('cpu(%)', []).append(p.cpu_percent()/psutil.cpu_count())
except:
continue
del p
return allprocesses
def updateInfo(data, table):
try:
table.clear()
for p in psutil.process_iter():
if p.pid in data['pid']:
try:
with p.oneshot():
data['memory(MB)'][data['pid'].index(p.pid)] = p.memory_full_info().uss/(1024**2)
data['memory(%)'][data['pid'].index(p.pid)] = p.memory_percent(memtype="rss")
data['cpu_times(s)'][data['pid'].index(p.pid)] = sum(p.cpu_times()[:2])
data['cpu(%)'][data['pid'].index(p.pid)] = p.cpu_percent()/psutil.cpu_count()
self.fillTable(data, len(data['pid']), len(data), table)
except:
continue
except:
pass
def tabledecorator(func):
#functools.wraps(func)
def wrapper(data, r, c, table):
func(data, r, c, table)
QTimer.singleShot(1000, lambda: self.updateInfo(data, table))
return wrapper
#tabledecorator
def fillTable(data, r, c, table):
table.setRowCount(r)
table.setColumnCount(c)
horHeaders = []
for n, key in enumerate(reversed(sorted(data.keys()))):
horHeaders.append(key)
for m, item in enumerate(data[key]):
newitem = QTableWidgetItem()
newitem.setData(Qt.DisplayRole, item)
table.setItem(m, n, newitem)
table.setHorizontalHeaderLabels(horHeaders)
table.resizeColumnsToContents()
table.resizeRowsToContents()
del horHeaders, n, key, m, item, newitem

There various performance issues in your implementation, but the most important is that fillTable is called for all items.
Since that function is decorated with the timer, the result is that you will be calling a delayed updateInfo for each row in the table, and since that function again calls the decorated fillTable, you're actually having a huge recursion problem: at every new cycle, the number of function calls grows exponentially.
If you have 2 matching processes, the first time updateInfo is called it will call fillTable two times while creating two QTimers. After one second you'll have two calls to updateInfo, resulting in 4 calls (2 processes multiplied by 2 calls to updateInfo); after another second they will be 8, then 16 and so on.
Another problem with your code is that, at each call to fillTable you're calling three functions that should only be executed once for each cycle:
setHorizontalHeaderLabels;
resizeColumnsToContents;
resizeRowsToContents;
The first one is pointless in any case, since the column labels will certainly not change through the lifetime of your program, and when that function is called it cycles through all its header items to check if any has changed.
The other two are very demanding in terms of performance, since the view is forced to query all items and call underlying functions to compute size hints of each row and column and then adapt the sizes accordingly.
There's really no point in using a single shot timer for this purpose, as the problem is that you're relying on function arguments to update the data, while a better approach is to properly use objects and references.
Other performance issues in your implementation:
since the keys of the dictionary are known and won't change, there's no point in using setdefault();
you're constantly retrieving the same lists at each cycle;
you're constructing the search list every time (which is time and memory consuming);
some values are clearly constants, there's no need in computing/retrieving them at every cycle;
A possible reimplementation and simplification of your code could be the following:
create a normal QTimer in the __init__, which updates the process list (which might change for obvious reasons) and update it;
create an empty dictionary with all keys set as empty lists;
set the table with the predefined column count and horizontal labels;
create a function that cycles through the processes, if one already exists, update its data, otherwise create a new row and append the new data;
optimize the function to improve its execution time and memory usage;
CpuCount = psutil.cpu_count()
MemoryRatio = 1024 ** 2
class ProcView(QtWidgets.QWidget):
def __init__(self):
super().__init__()
layout = QtWidgets.QVBoxLayout(self)
self.table = QtWidgets.QTableWidget(0, 6)
layout.addWidget(self.table)
self.headers = 'pid', 'memory(MB)', 'memory(%)', 'cpu_times(s)', 'create_time', 'cpu(%)'
self.updateColumns = 1, 2, 3, 5
self.table.setHorizontalHeaderLabels(self.headers)
self.procTimer = QtCore.QTimer(interval=1000, timeout=self.updateInfo)
self.filter = 'python3', 'python'
self.procs = {header:[] for header in self.headers}
self.procTimer.start()
self.updateInfo()
def updateInfo(self):
pids = self.procs['pid']
memoryMb = self.procs['memory(MB)']
memoryPerc = self.procs['memory(%)']
cpu_times = self.procs['cpu_times(s)']
create_times = self.procs['create_time']
cpuPerc = self.procs['cpu(%)']
for p in psutil.process_iter():
if not p.name().lower() in self.filter:
continue
with p.oneshot():
if p.pid not in pids:
row = len(pids)
self.table.insertRow(row)
pids.append(p.pid)
memoryMb.append(p.memory_full_info().uss / MemoryRatio)
memoryPerc.append(p.memory_percent(memtype="rss"))
cpu_times.append(sum(p.cpu_times()[:2]))
create_times.append(datetime.datetime.fromtimestamp(p.create_time()).strftime("%Y-%m-%d %H:%M:%S"))
cpuPerc.append(p.cpu_percent() / CpuCount)
for col, header in enumerate(self.headers):
item = QtWidgets.QTableWidgetItem()
item.setData(QtCore.Qt.DisplayRole, self.procs[header][row])
self.table.setItem(row, col, item)
else:
row = pids.index(p.pid)
memoryMb[row] = p.memory_full_info().uss / MemoryRatio
memoryPerc[row] = p.memory_percent(memtype="rss")
cpu_times[row] = sum(p.cpu_times()[:2])
cpuPerc[row] = p.cpu_percent() / CpuCount
for col in self.updateColumns:
item = self.table.item(row, col)
item.setData(QtCore.Qt.DisplayRole, self.procs[self.headers[col]][row])
self.table.resizeColumnsToContents()
self.table.resizeRowsToContents()
Note that a proper implementation should possibly:
avoid a dictionary with fields as keys and lists for each field, but eventually a dictionary with pid as keys and a dictionary for each item's field (a specialized class that works as an abstraction layer for the process would be even better);
use a custom model (with a QTableView);
verify whenever processes terminate;
use fixed row sizes and avoid automatic column resizing at each cycle for all columns (it's better to use the Fixed resize mode for common fixed sized columns like the CPU usage and leave the resizing to the user);

Multiple iterations of function with multiple arguments returning multiple values using Multiprocessing in python

I doing 100 iterations of the function model so, i tried using multiprocessing to distribute the tasks and for getting the final output I tried using queue but it takes too much time, failing the purpose of multiprocessing. How to solve this problem?
def model(X,Y):
ada_clf={}
pred1={}
auc_final=[]
for iteration in range(100):
ada_clf[iteration] = AdaBoostClassifier(DecisionTreeClassifier(),n_estimators=1000,learning_rate=0.001)
ada_clf[iteration].fit(X,Y)
pred1[iteration]=ada_clf[iteration].predict(test1)
individuallabelsfromada1=[]
for i in range(len(test1)):
individuallabelsfromada1.append([])
for j in range(100):
individuallabelsfromada1[i].append(pred1[j][i])
final_labels_ada1=[]
for each in individuallabelsfromada1:
final_labels_ada1.append(find_majority(each))
final=pd.Series(final_labels_ada1)
temp_arr=np.array(final)
total_labels2=pd.Series(temp_arr)
fpr, tpr, thresholds = roc_curve(y_test, total_labels2, pos_label=1)
auc_final.append(auc(fpr,tpr))
q.put(total_labels2)
q1.put(auc_final)
q2.put(ada_clf)
print('done')
overall_labels={}
final_auc={}
final_ada_clf={}
processes=[]
q=Queue()
q1=Queue()
q2=Queue()
for iteration in range(100):
if __name__=='__main__':
p=multiprocessing.Process(target=model,args=(x_train,y_labels,q,q1,q2,))
overall_labels[iteration]=q.get()
final_auc[iteration]=q1.get()
final_ada_clf[iteration]=q2.get()
p.start()
processes.append(p)
for each in processes:
each.join()
Below is my edited version, but returns only single output, i tried using multiple output but could not get it, so settled for only single output i.e. total_labels2:-
##code before this is same as before, only thing changed is arguments of model from def model(X,Y) to def model(repeat,X,Y)
total_labels2 = pd.Series(temp_arr)
return (repeat,total_labels2)
def get_result(total_labels2):
global testover_forall
testover_forall.append(total_labels2)
if __name__ == '__main__':
import multiprocessing as mp
testover_forall = []
pool = mp.Pool(40)
for repeat in range(100):
pool.apply_async(bound_model, args= repeat, x_train, y_train), callback= get_result)
pool.close()
pool.join()
repetations_index=[]
for i in range(100):
repetations_index.append(testover_forall[i][0])
final_last_labels = {}
for i in range(100):
temp = str(i)
final_last_labels[temp] = testover_forall[repetations_index[i]][1]
totally_last_labels=[]
for each in final_last_labels:
temp=np.array(final_last_labels[each])
totally_last_labels.append(temp)

See my comments (actually questions) to your post.
You should be using a multiprocessing pool to limit the number of processes that you create to the number of CPU cores that you have. This will also make it easier to get return values back from your model function instead of writing results to 3 different queues (and you could have written a tuple of 3 values to just one queue). You will, of course, require other import statements and code. Given your use of numpy and other libraries, which may be implemented in C Language, you could also retry running this using threading to see if that helps or hurts performance. Do this by replacing ProcessPoolExecutor with ThreadPoolExecutor in the two places it is referenced.
Note
Any changes that model makes to passed arguments X and Y will not be reflected back to the main process. So if model is called repeatedly with the same arguments over and over, as it appears to be, it's not clear whether each call will return different values, especially if the calls are being done in parallel.
from concurrent.futures import ProcessPoolExecutor
def model(X,Y):
ada_clf={}
pred1={}
auc_final=[]
for iteration in range(100):
ada_clf[iteration] = AdaBoostClassifier(DecisionTreeClassifier(),n_estimators=1000,learning_rate=0.001)
ada_clf[iteration].fit(X,Y)
pred1[iteration]=ada_clf[iteration].predict(test1)
individuallabelsfromada1=[]
for i in range(len(test1)):
individuallabelsfromada1.append([])
for j in range(100):
individuallabelsfromada1[i].append(pred1[j][i])
final_labels_ada1=[]
for each in individuallabelsfromada1:
final_labels_ada1.append(find_majority(each))
final=pd.Series(final_labels_ada1)
temp_arr=np.array(final)
total_labels2=pd.Series(temp_arr)
fpr, tpr, thresholds = roc_curve(y_test, total_labels2, pos_label=1)
auc_final.append(auc(fpr,tpr))
#q.put(total_labels2)
#q1.put(auc_final)
#q2.put(ada_clf)
return total_labels2, auc_final, ada_clf
#print('done')
if __name__ == '__main__':
with ProcessPoolExecutor() as executor:
futures = [executor.submit(model, x_train, y_labels) for iteration in range(100)]
# simple lists will suffice:
overall_labels = []
final_auc = []
final_ada_clf = []
for future in futures:
# get return value and store
total_labels2, auc_final, ada_clf = future.result()
overall_labels.append(total_labels2)
final_auc.append(auc_final)
final_ada_clf.append(ada_clf)
Update
It wasn't clear from the problem specification that the returned results are based on a random number generator and if successive calls to the worker function, model, do not employ a single random number generator across all processes in the multiprocessing pool, then the multiprocessing implementation will clearly return different results than the non-multiprocessing version. And it is not clear from the code provided where the random number generator is being used; it may be in library code that you have no access to. If that is the case, you have two options: (1) Use multithreading instead by changing the import statement as I have indicated in the code below; you may still achieve performance benefits as I have already mentioned or (2) Update the signature to model as follows. You will be passed a new argument, random_generator, that currently supports two methods, randint (like random.randint and random (like random.random), although it should be easy enough to modify the code if you need a different method from module random. You will use this random number generator in place of module random if you are able to. But note that this random generator will run much more slowly than the standard one; this is the price you pay.
Since we are also adding a repetition argument to model (it now has to be the final argument -- note the updated signature below), we can now use method map (no need to use a callback):
def model(X,Y, random_generator, repetition):
...
etc.
from multiprocessing import Pool
# or use the following import instead to use multithreading (but then use standard random generator):
# from multiprocessing.dummy import Pool
import random
from functools import partial
from multiprocessing.managers import BaseManager
class RandomGeneratorManager(BaseManager):
pass
class RandomGenerator:
def __init__(self):
random.seed(0)
def randint(self, a, b):
return random.randint(a, b)
def random(self):
return random.random()
# add other functions if needed
if __name__ == '__main__':
RandomGeneratorManager.register('RandomGenerator', RandomGenerator)
with RandomGeneratorManager() as manager:
random_generator = manager.RandomGenerator()
# why 40? why not use default, which is the number of cpu cores you have?:
pool = Pool(40):
worker = partial(model, x_train, y_labels, random_generator)
results = pool.map(worker, range(100))

Function that multiprocesses another function

I'm performing analyses of time-series of simulations. Basically, it's doing the same tasks for every time steps. As there is a very high number of time steps, and as the analyze of each of them is independant, I wanted to create a function that can multiprocess another function. The latter will have arguments, and return a result.
Using a shared dictionnary and the lib concurrent.futures, I managed to write this :
import concurrent.futures as Cfut
def multiprocess_loop_grouped(function, param_list, group_size, Nworkers, *args):
# function : function that is running in parallel
# param_list : list of items
# group_size : size of the groups
# Nworkers : number of group/items running in the same time
# **param_fixed : passing parameters
manager = mlp.Manager()
dic = manager.dict()
executor = Cfut.ProcessPoolExecutor(Nworkers)
futures = [executor.submit(function, param, dic, *args)
for param in grouper(param_list, group_size)]
Cfut.wait(futures)
return [dic[i] for i in sorted(dic.keys())]
Typically, I can use it like this :
def read_file(files, dictionnary):
for file in files:
i = int(file[4:9])
#print(str(i))
if 'bz2' in file:
os.system('bunzip2 ' + file)
file = file[:-4]
dictionnary[i] = np.loadtxt(file)
os.system('bzip2 ' + file)
Map = np.array(multiprocess_loop_grouped(read_file, list_alti, Group_size, N_thread))
or like this :
def autocorr(x):
result = np.correlate(x, x, mode='full')
return result[result.size//2:]
def find_lambda_finger(indexes, dic, Deviation):
for i in indexes :
#print(str(i))
# Beach = Deviation[i,:] - np.mean(Deviation[i,:])
dic[i] = Anls.find_first_max(autocorr(Deviation[i,:]), valmax = True)
args = [Deviation]
Temp = Rescal.multiprocess_loop_grouped(find_lambda_finger, range(Nalti), Group_size, N_thread, *args)
Basically, it is working. But it is not working well. Sometimes it crashes. Sometimes it actually launches a number of python processes equal to Nworkers, and sometimes there is only 2 or 3 of them running at a time while I specified Nworkers = 15.
For example, a classic error I obtain is described in the following topic I raised : Calling matplotlib AFTER multiprocessing sometimes results in error : main thread not in main loop
What is the more Pythonic way to achieve what I want ? How can I improve the control this function ? How can I control more the number of running python process ?

One of the basic concepts for Python multi-processing is using queues. It works quite well when you have an input list that can be iterated and which does not need to be altered by the sub-processes. It also gives you a good control over all the processes, because you spawn the number you want, you can run them idle or stop them.
It is also a lot easier to debug. Sharing data explicitly is usually an approach that is much more difficult to setup correctly.
Queues can hold anything as they are iterables by definition. So you can fill them with filepath strings for reading files, non-iterable numbers for doing calculations or even images for drawing.
In your case a layout could look like that:
import multiprocessing as mp
import numpy as np
import itertools as it
def worker1(in_queue, out_queue):
#holds when nothing is available, stops when 'STOP' is seen
for a in iter(in_queue.get, 'STOP'):
#do something
out_queue.put({a: result}) #return your result linked to the input
def worker2(in_queue, out_queue):
for a in iter(in_queue.get, 'STOP'):
#do something differently
out_queue.put({a: result}) //return your result linked to the input
def multiprocess_loop_grouped(function, param_list, group_size, Nworkers, *args):
# your final result
result = {}
in_queue = mp.Queue()
out_queue = mp.Queue()
# fill your input
for a in param_list:
in_queue.put(a)
# stop command at end of input
for n in range(Nworkers):
in_queue.put('STOP')
# setup your worker process doing task as specified
process = [mp.Process(target=function,
args=(in_queue, out_queue), daemon=True) for x in range(Nworkers)]
# run processes
for p in process:
p.start()
# wait for processes to finish
for p in process:
p.join()
# collect your results from the calculations
for a in param_list:
result.update(out_queue.get())
return result
temp = multiprocess_loop_grouped(worker1, param_list, group_size, Nworkers, *args)
map = multiprocess_loop_grouped(worker2, param_list, group_size, Nworkers, *args)
It can be made a bit more dynamic when you are afraid that your queues will run out of memory. Than you need to fill and empty the queues while the processes are running. See this example here.
Final words: it is not more Pythonic as you requested. But it is easier to understand for a newbie ;-)

Difference between map() and pool.map()

I have a code like this
def plotFrame(n):
a = data[n, :]
do_something_with(a)
data = loadtxt(filename)
ids = data[:,0] # some numbers from the first column of data
map(plotFrame, ids)
That worked fine for me. Now I want to try replacing map() with pool.map() as follows:
pools = multiprocessing.Pool(processes=1)
pools.map(plotFrame, ids)
But that won't work, saying:
NameError: global name 'data' is not defined
The questions is: What is going on? Why map() does not complain about the data variable that is not passed to the function, but pool.map() does?
EDIT:
I' m using Linux.
EDIT 2:
Based on #Bill 's second suggestion, I now have the following code:
def plotFrame_v2(line):
plot_with(line)
if __name__ == "__main__":
ff = np.loadtxt(filename)
m = int( max(ff[:,-1]) ) # max id
l = ff.shape[0]
nfig = 0
pool = Pool(processes=1)
for i in range(0, l/m, 50):
data = ff[i*m:(i+1)*m, :] # data of one frame contains several ids
pool.map(plotFrame_v2, data)
nfig += 1
plt.savefig("figs_bot/%.3d.png"%nfig)
plt.clf()
That works just as expected. However, now I have another unexpected problem: The produced figures are blank, whereas the above code with map() produces figures with the content of data.

Using multiprocessing.pool, you are spawning individual processes to work with the shared (global) resource data. Typically, you can allow the processes to work with a shared resource in the parent process by make that resource explicitly global. However, it is better practice to explicitly pass all needed resources to the child processes as function arguments. This is required if you are working on Windows. Check out the multiprocessing guidelines here.
So you could try doing
data = loadtxt(filename)
def plotFrame(n):
global data
a = data[n, :]
do_something_with(a)
ids = data[:,0] # some numbers from the first column of data
pools = multiprocessing.Pool(processes=1)
pools.map(plotFrame, ids)
or even better see this thread about feeding multiple arguments to a function with multiprocessing.pool. A simple way could be
def plotFrameWrapper(args):
return plotFrame(*args)
def plotFrame(n, data):
a = data[n, :]
do_something_with(a)
if __name__ == "__main__":
from multiprocessing import Pool
data = loadtxt(filename)
pools = Pool(1)
ids = data[:,0]
pools.map(plotFrameWrapper, zip([data]*len(inds), inds))
print results
One last thing: since it looks like the only thing you are doing from your example is slicing the array, you can simply slice first then pass the sliced arrays to your function:
def plotFrame(sliced_data):
do_something_with(sliced_data)
if __name__ == "__main__":
from multiprocessing import Pool
data = loadtxt(filename)
pools = Pool(1)
ids = data[:,0]
pools.map(plotFrame, data[ids])
print results

To avoid "unexpected" problems, avoid globals.
To reproduce your first code example with builtin map that calls plotFrame:
def plotFrame(n):
a = data[n, :]
do_something_with(a)
using multiprocessing.Pool.map, the first thing is to deal with the global data. If do_something_with(a) also uses some global data then it should also be changed.
To see how to pass a numpy array to a child process, see Use numpy array in shared memory for multiprocessing. If you don't need to modify the array then it is even simpler:
import numpy as np
def init(data_): # inherit data
global data #NOTE: no other globals in the program
data = data_
def main():
data = np.loadtxt(filename)
ids = data[:,0] # some numbers from the first column of data
pool = Pool(initializer=init, initargs=[data])
pool.map(plotFrame, ids)
if __name__=="__main__":
main()
All arguments either should be explicitly passed as arguments to plotFrame or inherited via init().
Your second code example tries to manipulate global data again (via plt calls):
import matplotlib.pyplot as plt
#XXX BROKEN, DO NOT USE
pool.map(plotFrame_v2, data)
nfig += 1
plt.savefig("figs_bot/%.3d.png"%nfig)
plt.clf()
Unless you draw something in the main process this code saves blank figures. Either plot in the child processes or send data to be plotted to the parent processes explicitly e.g., by returning it from plotFrame and using pool.map() returned value. Here's a code example: how to plot in child processes.