I try to use Multiprocessing in a class.I use Multiprocessing.pipe() to pass the instance o from parent process to child process.
self.listofwritetags = self.collectwritetaglist()
self.progressbar['value'] = 20
self.frame.update_idletasks()
self.alldevice = alldevices_V3.AllDevices(self.comm_object)
self.progressbar['value'] = 40
self.frame.update_idletasks()
con1,con2 = multiprocessing.Pipe()
con1.send(self.alldevice)
con2.send(self.comm_object)
# Multithreading section
# self.callmotor1dprocess = thread_with_trace(target=self.callallmotor1d,args=(self.comm_object,self.alldevice))
self.callmotor1dprocess = multiprocessing.Process(target = self.callallmotor1d,args= (con1,con2))
self.listofthread.append(self.callmotor1dprocess)
self.button2.config(text="Initialized")
initial = "True"
self.progressbar.stop()
Now I call all Multiprocessing Initiate
def startprocess(self):
for item in self.listofthread:
item.start()
self.button3.config(text="started")
def stopprocess(self):
for item in self.listofthread:
item.kill()
This Code I call inside the class.Now Executed method I should call outside of the class.
def callallmotor1d(con1,con2):
comobject = con1.recv()
devices = con2.recv()
while True:
Allmotorprocessing.process(comobject, devices)
But I got the error which is very common:-
Error message:-
Traceback (most recent call last):
File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
Traceback (most recent call last):
File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
I don't know why thread.lock object is created.
To avoid this error I try to modify my Alldevices class and comm_object class like this:-
Here is my modification:-
class AllDevices:
def __init__(self,comobject):
self.mylock = threading.Lock()
self.comobject = comobject
self.dfM1D = pd.read_excel(r'C:\OPCUA\Working_VF1_5.xls', sheet_name='Motor1D')
self.allmotor1dobjects = callallmotor1D_V3.Cal_AllMotor1D(self.dfM1D, self.comobject)
def __getstate__(self):
state = vars(self).copy()
# Remove the unpicklable entries.
del state['mylock']
return state
def __setstate__(self, state):
# Restore instance attributes.
vars(self).update(state)
Here is the comobject class.
class General():
def __init__(self):
self.client = Communication()
self.mylock = threading.Lock()
self.sta_con_plc = self.client.opc_client_connect()
self.readgeneral = ReadGeneral(self.client.PLC)
self.writegeneral = WriteGeneral(self.client.PLC)
def __getstate__(self):
state = vars(self).copy()
# Remove the unpicklable entries.
del state['mylock']
return state
def __setstate__(self, state):
# Restore instance attributes.
vars(self).update(state)
But still I got an error.
Is this my implementation is correct?
self.allmotor1dobjects = callallmotor1D_V2.Cal_AllMotor1D(self.dfM1D, self.comobject,self.logger)
Here self.allmotor1dobjects is also class instances.
like:-
self.client = Communication()
self.readgeneral = ReadGeneral(self.client.PLC)
self.writegeneral = WriteGeneral(self.client.PLC)
These are also class instances.
I never used thread.lock any of these two classes.
I don't know how it is created.
As per the docs suggested https://docs.python.org/3/library/pickle.html#pickling-class-instances
If I use getstate and setstate, it should remove this error.
In my case it doesn't work.
How can I remove this error.
any help in this regard will be highly appreciated
You can pass most classes through a pipe, but not if the instances have attributes that are unpicklable types. In this case, the instances of alldevice (the instances of which are stored in some collection you stored as self.devices) have, directly or indirectly, a threading.Lock attribute (the error message says _thread.lock because under all the abstraction, that's the actually class that threading.Lock returns an instance of).
threading.Lock isn't picklable (because thread locks only make sense within a given process; even if you recreated them in another process, they wouldn't actually provide any sort of synchronization between the processes).
If the alldevice class is under your control, you have a few options:
Remove the per-instance threading.Lock if possible (easiest solution, but assumes synchronization isn't required, or that synchronization could be done by a shared global lock)
If synchronization is required, and must operate across processes, but needn't be per-instance, and you're on a UNIX-like system (Linux, BSD) you could use a shared global multiprocessing.Lock() instead; it will be inherited when the Process call triggers a fork
If synchronization is required, and you must have a per-instance Lock (and it's okay for the lock to operate across processes), you can replace threading.Lock with a pickle-friendly multiprocessing.Manager's Lock. You'd need to make a common multiprocessing.Manager() instance somewhere (e.g. globally, or as a class attribute of alldevice), then use Lock from that manager instead of from threading. Simple example:
import multiprocessing
class alldevice:
MANAGER = multiprocessing.Manager() # Shared manager for all alldevice instances
def __init__(self, ... other args here ...):
self.mylock = self.MANAGER.Lock() # Make picklable lock
# ... rest of initialization ...
When multiprocessing isn't involved, this will be slower than a threading.Lock, as it will require IPC to lock and unlock the lock (the Manager's Lock instances are actually a proxy object that communicates with the Manager on whatever process it's actually running in), and it will lock across processes (probably for the best if you're locking access to actual hardware that can't be used concurrently from multiple processes), but it's relatively simple.
If synchronization is required, but only within a process, not across processes, you can take control of the pickling process to avoid trying to pickle the threading.Lock, and instead recreate it when it's unpickled on the other side. Just explicitly implement the pickle support methods to avoid pickling the Lock, and force it to be recreated on the other side. Example:
import copy
class alldevice:
def __init__(self, ... other args here ...):
self.mylock = threading.Lock() # Use regular lock
# ... rest of initialization ...
def __getstate__(self):
state = vars(self).copy() # Make copy of instance dict
del state['mylock'] # Remove lock attribute
return state
def __setstate__(self, state):
vars(self).update(state)
self.mylock = threading.Lock() # Make new lock on other side
# If you ever make copies within the same thread, you may want to define
# __deepcopy__ so the local copy process doesn't make a new lock:
def __deepcopy__(self, memo):
# Make new empty instance without invoking __init__
newself = self.__class__.__new__(self.__class__)
# Individually deepcopy attributes *except* mylock, which we alias
for name, value in vars(self).items():
# Cascading deepcopy for all other attributes
if name != 'mylock':
value = copy.deepcopy(value, memo)
setattr(newself, name, value)
return newself
The __deepcopy__ override is only needed if you want the copy to continue sharing the lock; otherwise, if a deep copy should behave as an entirely independent instance, you can omit it, and you'll end up with an unrelated lock in the copy.
If you don't have control of the alldevice class, but can identify the problematic attribute, your only option is to register a copyreg handler for alldevice to do the same basic thing as option #4, which would look something like this:
import copyreg
def unpickle_alldevice(state):
self = alldevice.__new__(alldevice) # Make empty alldevice
vars(self).update(state) # Update with provided state
self.mylock = threading.Lock() # Make fresh lock
return self
def pickle_alldevice(ad):
state = vars(ad).copy() # Make shallow copy of instance dict
del state['mylock'] # Remove lock attribute
return unpickle_alldevice, (state,) # Return __reduce__ style info for reconstruction
# Register alternate pickler for alldevice
copyreg.pickle(alldevice, pickle_alldevice)
Related
I am running the below code
from multiprocessing import Process
from multiprocessing import Value
class TestMultiprocess(Process):
def __init__(self):
super().__init__()
self.variable={"value":"initial"}
def run(self):
print("executing the run function")
self.variable={"value":"new"}
t=TestMultiprocess()
t.start()
t.join()
print(t.variable) #prints {"value":"initial"}, (I expected {"value":"new"})
I am setting the value of the variable in the run function but this value is not getting reflected in the object (It still holds the value initialised in constructor). Why is this??
How do I set/change am attribute value in the run function of the class inheriting multiprocessing and use the value from the client code which creates the object of the class. I looked at multiprocessing.values but this does not have support for dictionaries. The attribute value I need to set is a dictionary
Your process is running in a different address space. Therefore, your TestMultiProcess instance must be serialized and sent to that child process's address space where it is then deserialized (all of this is done with the pickle module). So what is being updated is a copy of your instance that is in the child process's address space; the main process's instance is never updated.
The simplest solution is to used a managed dictionary. The actual dictionary "lives" in the address space of the Manager and variable managed_dict in the code below is actually a proxy object such that when one of its methods is called, the method name and its arguments are sent to the Manager process where the actual dictionary is updated:
from multiprocessing import Process, Manager
class TestMultiprocess(Process):
def __init__(self, managed_dict):
super().__init__()
self.managed_dict = managed_dict
self.managed_dict["value"] = "initial"
def run(self):
print("executing the run function")
self.managed_dict["value"] = "new"
# Required for Windows:
if __name__ == '__main__':
with Manager() as manager:
managed_dict = manager.dict()
t = TestMultiprocess(managed_dict)
print(t.managed_dict)
t.start()
t.join()
print(t.managed_dict)
Prints:
{'value': 'initial'}
executing the run function
{'value': 'new'}
I have a large Python 3.6 system where multiple processes and threads interact with each other and the user. Simplified, there is a Scheduler instance (subclasses threading.Thread) and a Worker instance (subclasses multiprocessing.Process). Both objects run for the entire duration of the program.
The user interacts with the Scheduler by adding Task instances and the Scheduler passes the task to the Worker at the correct moment in time. The worker uses the information contained in the task to do its thing.
Below is some stripped out and simplified code out of the project:
class Task:
def __init__(self, name:str):
self.name = name
self.state = 'idle'
class Scheduler(threading.Thread):
def __init__(self, worker:Worker):
super().init()
self.worker = worker
self.start()
def run(self):
while True:
# Do stuff until the user schedules a new task
task = Task() # <-- In reality the Task intance is not created here but the thread gets it from elsewhere
task.state = 'scheduled'
self.worker.change_task(task)
# Do stuff until the task.state == 'finished'
class Worker(multiprocessing.Process):
def __init__(self):
super().init()
self.current_task = None
self.start()
def change_task(self, new_task:Task):
self.current_task = new_task
self.current_task.state = 'accepted-idle'
def run(self):
while True:
# Do stuff until the current task is updated
self.current_task.state = 'accepted-running'
# Task is running
self.current_task.state = 'finished'
The system used to be structured so that the task contained multiple multiprocessing.Events indicating each of its possible states. Then, not the whole Task instance was passed to the worker, but each of the task's attributes was. As they were all multiprocessing safe, it worked, with a caveat. The events changed in worker.run had to be created in worker.run and back passed to the task object for it work. Not only is this a less than ideal solution, it no longer works with some changes I am making to the project.
Back to the current state of the project, as described by the python code above. As is, this will never work because nothing makes this multiprocessing safe at the moment. So I implemented a Proxy/BaseManager structure so that when a new Task is needed, the system gets it from the multiprocessing manager. I use this structure in a sightly different way elsewhere in the project as well. The issue is that the worker.run never knows that the self.current_task is updated, it remains None. I expected this to be fixed by using the proxy but clearly I am mistaken.
def Proxy(target: typing.Type) -> typing.Type:
"""
Normally a Manager only exposes only object methods. A NamespaceProxy can be used when registering the object with
the manager to expose all the attributes. This also works for attributes created at runtime.
https://stackoverflow.com/a/68123850/8353475
1. Instead of exposing all the attributes manually, we effectively override __getattr__ to do it dynamically.
2. Instead of defining a class that subclasses NamespaceProxy for each specific object class that needs to be
proxied, this method is used to do it dynamically. The target parameter should be the class of the object you want
to generate the proxy for. The generated proxy class will be returned.
Example usage: FooProxy = Proxy(Foo)
:param target: The class of the object to build the proxy class for
:return The generated proxy class
"""
# __getattr__ is called when an attribute 'bar' is called from 'foo' and it is not found eg. 'foo.bar'. 'bar' can
# be a class method as well as a variable. The call gets rerouted from the base object to this proxy, were it is
# processed.
def __getattr__(self, key):
result = self._callmethod('__getattribute__', (key,))
# If attr call was for a method we need some further processing
if isinstance(result, types.MethodType):
# A wrapper around the method that passes the arguments, actually calls the method and returns the result.
# Note that at this point wrapper() does not get called, just defined.
def wrapper(*args, **kwargs):
# Call the method and pass the return value along
return self._callmethod(key, args, kwargs)
# Return the wrapper method (not the result, but the method itself)
return wrapper
else:
# If the attr call was for a variable it can be returned as is
return result
dic = {'types': types, '__getattr__': __getattr__}
proxy_name = target.__name__ + "Proxy"
ProxyType = type(proxy_name, (NamespaceProxy,), dic)
# This is a tuple of all the attributes that are/will be exposed. We copy all of them from the base class
ProxyType._exposed_ = tuple(dir(target))
return ProxyType
class TaskManager(BaseManager):
pass
TaskProxy = Proxy(Task)
TaskManager.register('get_task', callable=Task, proxytype=TaskProxy)
I would like to create shared object among processes. First I created server process which spawned process for class ProcessClass. Then I created another process where I want to connect to shared object.
But connection from another process created its own instance of ProcessClass.
So what I need to do to access this remote shared object.
Here is my test code.
from multiprocessing.managers import BaseManager
from multiprocessing import Process
class ProcessClass:
def __init__(self):
self._state = False
def set(self):
self._state = True
def get(self):
return self._state
class MyManager(BaseManager):
pass
def another_process():
MyManager.register('my_object')
m = MyManager(address=('', 50000))
m.connect()
proxy = m.my_object()
print(f'state from another process: {proxy.get()}')
def test_spawn_and_terminate_process():
MyManager.register('my_object', ProcessClass)
m = MyManager(address=('', 50000))
m.start()
proxy = m.my_object()
proxy.set()
print(f'state from main process: {proxy.get()}')
p = Process(target=another_process)
p.start()
p.join()
print(f'state from main process: {proxy.get()}')
if __name__ == '__main__':
test_spawn_and_terminate_process()
Output is
python test_communication.py
state from main process: True
state from another process: False
state from main process: True
Your code is working as it is supposed to. If you look at the documentation for multiprocessing.managers.SyncManager you will see that there is, for example, a method dict() to create a shareable dictionary. Would you expect that calling this method multiple times would return the same dictionary over and over again or new instances of sharable dictionaries?
What you need to do is enforce a singleton instance to be used repeatedly for successive invocations of proxy = m.my_object() and the way to do that is to first define the following function:
singleton = None
def get_singleton_process_instance():
global singleton
if singleton is None:
singleton = ProcessClass()
return singleton
Then you need to make a one line change in funtion test_spawn_and_terminate_process:
def test_spawn_and_terminate_process():
#MyManager.register('my_object', ProcessClass)
MyManager.register('my_object', get_singleton_process_instance)
This ensures that to satisfy requests for 'my_object', it always invokes get_singleton_process_instance() (returning the singleton) instead of ProcessClass(), which would return a new instance.
I'm trying to understand why python can not compile the following class.
class SharedResource(multiprocessing.Lock):
def __init__(self, blocking=True, timeout=-1):
# super().__init__(blocking=True, timeout=-1)
self.blocking = blocking
self.timeout = timeout
self.data = {}
TypeError: method expected 2 arguments, got 3
The reason why I'm subclassing Lock
my objective is to create a shared list of resource that should be usable only by on process at a time.
this concept will be eventually in a Flash application where the request should not be able to use the resource concurrently
RuntimeError: Lock objects should only be shared between processes through inheritance
class SharedResource():
def __init__(self, id, model):
'''
id: mode id
model: Keras Model only one worker at a time can call predict
'''
self.mutex = Lock()
self.id = id
self.model = model
manager = Manager()
shared_list = manager.list() # a List of models
shared_list.append(SharedResource())
def worker1(l):
...read some data
while True:
resource = l[0]
with m:
resource['model'].predict(...some data)
time.sleep(60)
if __name__ == "__main__":
processes = [ Process(target=worker1, args=[shared_list])]
for p in processes:
p.start()
for p in processes:
p.join()
The reason you are getting this error is because multiprocessing.Lock is actually a function.
In .../multiprocessing/context.py there are these lines:
def Lock(self):
'''Returns a non-recursive lock object'''
from .synchronize import Lock
return Lock(ctx=self.get_context())
This may change in the future so you can verify this on your version of python by doing:
import multiprocessing
print(type(multiprocessing.Lock))
To actually subclass Lock you will need to do something like this:
from multiprocessing import synchronize
from multiprocessing.synchronize import Lock
# Since Lock is now a class, this should work:
class SharedResource(Lock):
pass
I'm not endorsing this approach as a "good" solution, but it should solve your problem if you really need to subclass Lock. Subclassing things that try to avoid being subclassed is usually not a great idea, but sometimes it can be necessary. If you can solve the problem in a different way you may want to consider that.
I create Singleton class using Metaclass, it working good in multithreadeds and create only one instance of MySingleton class but in multiprocessing, it creates always new instance
import multiprocessing
class SingletonType(type):
# meta class for making a class singleton
def __call__(cls, *args, **kwargs):
try:
return cls.__instance
except AttributeError:
cls.__instance = super(SingletonType, cls).__call__(*args, **kwargs)
return cls.__instance
class MySingleton(object):
# singleton class
__metaclass__ = SingletonType
def __init__(*args,**kwargs):
print "init called"
def task():
# create singleton class instance
a = MySingleton()
# create two process
pro_1 = multiprocessing.Process(target=task)
pro_2 = multiprocessing.Process(target=task)
# start process
pro_1.start()
pro_2.start()
My output:
init called
init called
I need MySingleton class init method get called only once
Each of your child processes runs its own instance of the Python interpreter, hence the SingletonType in one process doesn't share its state with those in another process. This means that a true singleton that only exists in one of your processes will be of little use, because you won't be able to use it in the other processes: while you can manually share data between processes, that is limited to only basic data types (for example dicts and lists).
Instead of relying on singletons, simply share the underlying data between the processes:
#!/usr/bin/env python3
import multiprocessing
import os
def log(s):
print('{}: {}'.format(os.getpid(), s))
class PseudoSingleton(object):
def __init__(*args,**kwargs):
if not shared_state:
log('Initializating shared state')
with shared_state_lock:
shared_state['x'] = 1
shared_state['y'] = 2
log('Shared state initialized')
else:
log('Shared state was already initalized: {}'.format(shared_state))
def task():
a = PseudoSingleton()
if __name__ == '__main__':
# We need the __main__ guard so that this part is only executed in
# the parent
log('Communication setup')
shared_state = multiprocessing.Manager().dict()
shared_state_lock = multiprocessing.Lock()
# create two process
log('Start child processes')
pro_1 = multiprocessing.Process(target=task)
pro_2 = multiprocessing.Process(target=task)
pro_1.start()
pro_2.start()
# Wait until processes have finished
# See https://stackoverflow.com/a/25456494/857390
log('Wait for children')
pro_1.join()
pro_2.join()
log('Done')
This prints
16194: Communication setup
16194: Start child processes
16194: Wait for children
16200: Initializating shared state
16200: Shared state initialized
16201: Shared state was already initalized: {'x': 1, 'y': 2}
16194: Done
However, depending on your problem setting there might be better solutions using other mechanisms of inter-process communication. For example, the Queue class is often very useful.