Develop Reference

Shared memory in multiprocessing.pool

Code:
import multiprocessing
import multiprocessing.managers
class A:
name = "A name"
arr_b = []
def __init__(self, num):
for i in range(5):
self.arr_b.append(B())
class B:
name = "B name"
def __init__(self):
pass
def func(num):
return A(num)
if __name__ == '__main__':
pool = multiprocessing.Pool()
result = pool.map(func, range(5))
for res in result:
print(res.name)
print(res.arr_b)
Result:
A name
[]
A name
[]
A name
[]
A name
[]
A name
[]
How i can share array of B class objects correctly?
I tried to used Manager and BaseManager, but it allows me to use created in main object. But i need to create object in func and return it into main.

Python multiprocessing, share class instance does not work

I want to send tasks to the POOL inside the shared class based on some conditions. But I got some unexpected result, Which are shown below.
• Why the len(self.map) is 0, not 100.
• Do I have to reconstruct my code to achieve this goal.
from multiprocessing import Pool
from multiprocessing.managers import BaseManager
pool = None
def doSomething(obj, *args):
obj.doSomething(*args)
class SharedClass:
def __init__(self):
global pool
self.map = set()
pool = Pool(4)
def someCondition(self):
# the condition is rely on the instance, here is just an example
return True
def go(self, n):
global pool
for i in xrange(n):
if self.someCondition():
# pass the shared class to other process
pool.apply_async(doSomething, (self, i))
pool.close()
pool.join()
# got AssertionError here
# why the len of self.map is 0
assert len(self.map) == 100
def doSomething(self, n):
# this should change the same SharedClass instance?
self.map.add(n)
class MyManager(BaseManager):
pass
MyManager.register("SharedClass", SharedClass)
def main():
manager = MyManager()
manager.start()
obj = manager.SharedClass()
obj.go(100)
if __name__ == "__main__":
main()

Python multiprocessing: Functions called using pool cannot access global variables declared inside if '__name__' == '__main__' bock

I am trying to access a global variable declared inside if __name__ == '__main__' block from a function which is called by object of class Pool for multiprocessing purpose.
But it throws NameError.
But if I place same variables outside the block the code works just fine.
Demo:
When declared outside if __name__ == '__main__' block
from multiprocessing import Pool
foo = 'OKAY'
def f(x):
print(foo)
return x*x
if __name__ == '__main__':
p = Pool(2)
p.map(f, [1, 2])
Output:
OKAY
OKAY
When declared inside if __name__ == '__main__':
from multiprocessing import Pool
def f(x):
print(foo)
return x*x
if __name__ == '__main__':
foo = 'OKAY'
p = Pool(2)
p.map(f, [1, 2])
Output:
NameError: name 'foo' is not defined

Scope of variables (with threads)

I have the following code in two modules.
Module "main":
#!/usr/bin/python
import threading
import b
a = 0
def api_a():
global a
print("api_a()")
a = 1
def main():
global a
thread_B = b.B_thread()
print("a = " + str(a))
thread_B.start()
# api_a()
thread_B.join()
print("a = " + str(a))
if __name__ == '__main__':
main()
Module "B":
#!/usr/bin/python
import threading
import main
class B_thread (threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
print("Starting " + self.name)
B_process()
print("Exiting " + self.name)
def B_process():
main.api_a()
If I run this code I get:
a = 0
Starting Thread-1
api_a()
Exiting Thread-1
a = 0
Why is variable "a" not set to "1" ?
If I activate the commented line of code in Module "main" ( api_a() ) the variable "a" will be set to "1". Why is the variable not set when the function api_a() is called via the thread?
In the example code I skipped the use of locks to make the code thread-safe.
Any ideas?
Thanks a lot in advance,
Thomas

Such behavior occurs, because you passed main.py as the argument to python (python main.py).
When you execute a script in such way, python interprets main module as __main__, but b module still updates main.a variable instead of __main__.a, because b has re-imported your main.py module as main.
So to make it work, we can either update __main__.a in api_a function from perspective of b module OR print main.a variable instead of __main__.a from perspective of __main__ module.
def api_a():
print("api_a()")
import sys
sys.modules['__main__'].a = 1
or
def main():
...
thread_B.join()
import sys
print("a = " + str(sys.modules['main'].a))
__main__ docs

simple passing of variables to classes

Why doesn't this work ?
class spin(threading.Thread):
def __init__(self):
super(spin,self).__init__()
self._stop = False
def run (self,var):
self.var=var
print self.var
def echo (self,var):
self.var=var
print self.var
if __name__ == '__main__':
s = spin()
s.start('hello')
but this does?
s = spin()
s.echo('hello')
Im guessing its because the start sequence needs to be defined in the init ? but not sure how. someone asked for error codes from this:
s.start('hello')
TypeError: start() takes exactly 1 argument (2 given)

The reason s.start('hello') does not work is that the inherited Threadin.start() method takes no arguments other than self.
Calling s.echo('hello') does appear to work, but it calls the function in the context of the main thread instead of spawning a new thread.
One way to fix your code is by supplying var to the constructor:
import threading
class Spin(threading.Thread):
def __init__(self,var):
super(Spin,self).__init__()
self.var = var
def run(self):
print self.var
if __name__ == '__main__':
s = Spin('hello')
s.start()
s.join()