I'm trying to use multiprocessing for doing multiple background jobs and using the main process as user interface which accepts commands through input(). Each process has to do certain jobs and will write its current status into a dictionary, which was created with manager.dict() and then passed to the Process.
After the creation of the processes, there is a loop with an input() for accessing the user commands. The commands are reduced to a minimum for simplicity here.
from multiprocessing import Manager
from multiprocessing import Process
with Manager() as manager:
producers = []
settings = [{'name':'test'}]
for setting in settings:
status = manager.dict()
logger.info("Start Producer {0}".format(setting['name']))
producer = Process(target=start_producer, args=(setting, status))
producer.start()
producers.append([producer, status])
logger.info("initialized {0} producers".format(len(producers)))
while True:
text_command = input('Enter your command:')
if text_command == 'exit':
logger.info("waiting for producers")
for p in producers:
p[0].join()
logger.info("Exit application.")
break
elif text_command == 'status':
for p in producers:
if 'name' in p[1] and 'status' in p[1]:
print('{0}:{1}'.format(p[1]['name'], p[1]['status']))
else:
print("Unknown command.")
The method which runs in other processes is pretty simple:
def start_producer(producer_setting: dict, status_dict: dict):
importer = MyProducer(producer_setting)
importer.set_status_dict(status_dict)
importer.run()
I create a MyProducer instance and set the status-dictionary through a setter of the object and call the blocking run() method, which will only return when the producer is finished. On calling set_status_dict(status_dict), the dictionary is filled with a name and status element.
When I run the code, the producer seems to get created, I receive the "Start Producer test" and "initialized 1 producers" output and after that the "Enter your command" request from the input(), but it seems that the actual process doesn't run.
When I press enter to skip the first loop iteration, I get the expected "unknown command" log and the producer-process begins the actual work. After that my "status" command also works as expected.
When I enter 'status' in the first loop-iteration I get an key-Error, because 'name' and 'status' are not set in the dictionary. Those keys should get set in set_status_dict() which itself is called in Process(target=...).
Why is that? Shouldn't producer.start() run the complete block of start_producer inside a new process and therefor never hang on the input() of the main-process?
How can I start the processes first without any user input and only then wait for input()?
Edit: A complete mvce programm with this problem can be found here: https://pastebin.com/k8xvhLhn
Edit: A solution with sleep(1) after initializing the processes has been found. But why does that behavior happen in the first place? Shouldn't run all code in start_producer() run in a new process?
I have limited experience with the multiprocessing module but I was able to get it to behave the way (i think) you want. First I added some print statements at the top of the while loop to see what might be going on and found that if the process was run or joined it worked. I figured you didn't want it to block so I added the call to run further up the process - but it appears that run() also blocks. Turns out that the process just wasn't finished when the first while loop iteration came around - adding time.sleep(30) at the top of the loop gave the process enough time to get scheduled (by the OS) and run. (On my machine it actually only needs between 200 and 300 milliseconds of nap time)
I replaced start_producer with :
def start_producer(producer_setting: dict, status_dict: dict):
## importer = MyProducer(producer_setting)
## importer.set_status_dict(status_dict)
## importer.run()
#time.sleep(30)
status_dict['name'] = 'foo'
status_dict['status'] = 'thinking'
Your code modified:
if __name__ == '__main__':
with Manager() as manager:
producers = []
settings = [{'name':'test'}]
for setting in settings:
status = manager.dict()
logger.info("Start Producer {0}".format(setting['name']))
producer = Process(target=start_producer, args=(setting, status))
producer.start()
# add a call to run() but it blocks
#producer.run()
producers.append([producer, status])
logger.info("initialized {0} producers".format(len(producers)))
while True:
time.sleep(30)
for p, s in producers:
#p.join()
#p.run()
print(f'name:{p.name}|alive:{p.is_alive()}|{s}')
if 'name' in s and 'status' in s:
print('{0}:{1}'.format(s['name'], s['status']))
text_command = input('Enter your command:')
if text_command == 'exit':
logger.info("waiting for producers")
for p in producers:
p[0].join()
logger.info("Exit application.")
break
elif text_command == 'status':
for p in producers:
if 'name' in p[1] and 'status' in p[1]:
print('{0}:{1}'.format(p[1]['name'], p[1]['status']))
else:
print("Unknown command.")
Related
I have a python script which calls a series of sub-processes. They need to run "for ever" - but they occasionally die, or get killed. When this happens I need to restart the process using the same arguments as the one which died.
This is a very simplified version:
[edit: this is the less simplified version, which includes "restart" code]
import multiprocessing
import time
import random
def printNumber(number):
print("starting :", number)
while random.randint(0, 5) > 0:
print(number)
time.sleep(2)
if __name__ == '__main__':
children = [] # list
args = {} # dictionary
for processNumber in range(10,15):
p = multiprocessing.Process(
target=printNumber,
args=(processNumber,)
)
children.append(p)
p.start()
args[p.pid] = processNumber
while True:
time.sleep(1)
for n, p in enumerate(children):
if not p.is_alive():
#get parameters dead child was started with
pidArgs = args[p.pid]
del(args[p.pid])
print("n,args,p: ",n,pidArgs,p)
children.pop(n)
# start new process with same args
p = multiprocessing.Process(
target=printNumber,
args=(pidArgs,)
)
children.append(p)
p.start()
args[p.pid] = pidArgs
I have updated the example to illustrate how I want the processes to be restarted if one crashes/killed/etc - keeping track of which pid was started with which args.
Is this the "best" way to do this, or is there a more "python" way of doing this?
I think I would create a separate thread for each Process and use a ProcessPoolExecutor. Executors have a useful function, submit, which returns a Future. You can wait on each Future and re-launch the Executor when the Future is done. Arguments to the function are tracked as class variables, so restarting is just a simple loop.
import threading
from concurrent.futures import ProcessPoolExecutor
import time
import random
import traceback
def printNumber(number):
print("starting :", number)
while random.randint(0, 5) > 0:
print(number)
time.sleep(2)
class KeepRunning(threading.Thread):
def __init__(self, func, *args, **kwds):
self.func = func
self.args = args
self.kwds = kwds
super().__init__()
def run(self):
while True:
with ProcessPoolExecutor(max_workers=1) as pool:
future = pool.submit(self.func, *self.args, **self.kwds)
try:
future.result()
except Exception:
traceback.print_exc()
if __name__ == '__main__':
for process_number in range(10, 15):
keep = KeepRunning(printNumber, process_number)
keep.start()
while True:
time.sleep(1)
At the end of the program is a loop to keep the main thread running. Without that, the program will attempt to exit while your Processes are still running.
For the example you provided I would just remove the exit condition from the while loop and change it to True.
As you said though the actual code is more complicated (why didn't you post that?). So if the process gets terminated by lets say an exception just put the code inside a try catch block. You can then put said block in an infinite loop.
I hope this is what you are looking for but that seems to be the right way to do it provided the goal and information you provided.
Instead of just starting the process immediately, you can save the list of processes and their arguments, and create another process that checks they are alive.
For example:
if __name__ == '__main__':
process_list = []
for processNumber in range(5):
process = multiprocessing.Process(
target=printNumber,
args=(processNumber,)
)
process_list.append((process,args))
process.start()
while True:
for running_process, process_args in process_list:
if not running_process.is_alive():
new_process = multiprocessing.Process(target=printNumber, args=(process_args))
process_list.remove(running_process, process_args) # Remove terminated process
process_list.append((new_process, process_args))
I must say that I'm not sure the best way to do it is in python, you may want to look at scheduler services like jenkins or something like that.
I need to run and close thread with input. Here, it uses input function to handle input. If input is start. then, run recv method as thread else input is close set second argument to False. False value to indicate loop inside recv to stop. But, My console keep open. It should close console. Cause it end of loop.
class Device():
def open(self, port, baudrate):
try:
return serial.Serial(port, baudrate)
except SerialException as e:
error = re.findall(r"'(.*?)'", str(e))
self.__error['port'] = error[0]
self.__error['description'] = error[1]
return None
def __state(self, open):
if open is None:
if self.__error['description'] == 'Access is denied.':
return True
elif self.__error['description'] == 'Port is already open.':
return True
else:
return False
else:
return True
def write(self, device, command):
if self.__state(device):
device.write(command.encode('UTF-8') + b'\r')
else:
print(self.__error['port'] + ' ' + self.__error['description'])
def recv(self, device, open = True):
while open:
if self.__state(device):
buffer = device.readline()
print(buffer)
time.sleep(1)
else:
print(device[0] + ' ' + device[1])
time.sleep(1)
device = Device()
serial = device.open('COM12', 9600)
while True:
command = input('Enter a command: ')
if command == 'start':
t = threading.Thread(target=device.recv, args=(serial,))
t.start()
elif command == 'close':
device.recv(serial, False)
elif command == 'imei':
device.write(serial, 'AT+CGSN')
If I am understanding your question, you are trying to do this:
When you begin the program and issue the start command, a new thread (Thread-2) begins executing in the recv() method of the device object
The main thread (Thread-1) continues in the event loop and presents another input prompt. At the same time, Thread-2 is looping forever inside of recv()
In Thread-1, you then issue the close command with the intention that it should disrupt Thread-2, causing it to break out of the loop and stop writing output to the terminal.
If that is correct, then the problem is in step 3: When Thread-1 makes the call device.recv(serial, False), it has no impact on the execution of Thread-2. That is because, when a thread is created and the start method is called:
t = threading.Thread(target=my_func)
t.start()
the new thread will begin executing my_func, but it will do so with it's own call stack, instruction pointer, and registers. Later, when Thread-1 calls device.recv(), it will result in a stack frame being pushed onto the call stack of Thread-1. That call is completely separate from the one that is already running in Thread-2.
Threads share many resources: Text, Data and BSS memory segments; open file descriptors; and signals (plus some other resources as well). Call stacks are not shared1.
If you need to communicate between threads, the threading library provides several options that can help. One such option is threading.Event, which can be used to communicate the occurrence of some event between threads. You could use it like this:
term_event = threading.Event()
class Device:
def recv(self, device):
while not term_event.isSet():
if self.__state(device):
buffer = device.readline()
print(buffer)
time.sleep(1)
else:
print(device[0] + ' ' + device[1])
time.sleep(1)
term_event.clear()
This creates an event object that is shared by all of the threads in the process. It can be used by Thread-1 as a way to tell Thread-2 when to exit. Then, you need to change the event loop like this:
while True:
command = input('Enter a command: ')
if command == 'start':
t = threading.Thread(target=device.recv, args=(serial,))
t.start()
elif command == 'close':
term_event.set()
elif command == 'imei':
device.write(serial, 'AT+CGSN')
Instead of calling the recv method a second time, juist set the shared event object, which Thread-2 interprets as a signal to exit. Before exiting, Thread-2 calls term_event.clear(), so a new thread can be started later on.
1: Since the threads are part of the same process, they actually occupy the same memory space as allocated by the kernel. As a consequence, each thread's private stack is (in theory) accessible by any other thread in that program.
I have long running process, that I want to keep track about in which state it currently is in. There is N processes running in same time therefore multiprocessing issue.
I pass Queue into process to report messages about state, and this Queue is then read(if not empty) in thread every couple of second.
I'm using Spider on windows as environment and later described behavior is in its console. I did not try it in different env.
from multiprocessing import Process,Queue,Lock
import time
def test(process_msg: Queue):
try:
process_msg.put('Inside process message')
# process...
return # to have exitstate = 0
except Exception as e:
process_msg.put(e)
callback_msg = Queue()
if __name__ == '__main__':
p = Process(target = test,
args = (callback_msg,))
p.start()
time.sleep(5)
print(p)
while not callback_msg.empty():
msg = callback_msg.get()
if type(msg) != Exception:
tqdm.write(str(msg))
else:
raise msg
Problem is that whatever I do with code, it never reads what is inside the Queue(also because it never puts anything in it). Only when I switch to dummy version, which runs similary to threading on only 1 CPU from multiprocessing.dummy import Process,Queue,Lock
Apparently the test function have to be in separate file.
I have a logging filter that checks for an environment variable to change and I want it to run (in the background) in a thread separate from the process that is setting the environment variable.
What I'm trying to do: every time logging.ERROR is called in my code, the user is alerted to the error and prompted on whether or not they want to continue. Separately the filter and the prompt work correctly however, when I put them together I have a problem. I need to have the filter running in the background so the code to prompt the user can run simultaneously (right now, the filter executes first and the prompt shows up after the while loop in the filter times out, at which point it is useless).
My filter code:
class ErrorFilter(logging.Filter):
def __init__(self,level):
self.level = level
thread = threading.Thread(target=self.filter,args=())
thread.daemon = True
thread.start()
def filter(self,record):
if record.levelno == self.level:
os.environ["ERROR_FLAG"] = "True"
timeout = time.time() + 60*1 #set the timeout to 1 minute
while True:
print "waiting..."
keep_going = os.environ.get("CONTINUE_FLAG")
#wait for user to respond
if keep_going == "False" or time.time() > timeout:
print "oops there's a problem, quitting."
break
if keep_going == "True":
print "Continuing"
break
os.environ["CONTINUE_FLAG"] = "None"
I have another short method that "listens" for ERROR_FLAG and then asks for input using:
def continueAsk(message, title="Warning! Continue?", yn=("Yes","No")):
yes = set(['yes','y', 'ye', '', 'canhaz'])
no = set(['no','n', 'lolzno'])
tryLimit = 0
while tryLimit < 100:
sys.stdout.write(message + ": ")
choice = raw_input().lower()
if choice in yes:
return True
elif choice in no:
return False
else:
tryLimit+=1
sys.stdout.write("Please respond with 'yes' or 'no'.")
EDIT
I've also tried using multiprocessing in my filter like this:
from multiprocessing import Process, Queue
def __init__(self,level):
self.level = level
queue = Queue()
p = Process(target=self.filter,args=("hi"))
p.start()
p.join()
I've tried setting up my filter so it runs in a different thread, but I've not had any luck so far (the filter still runs first, followed by the prompt) and I've never used multithreading before. I know this is not a traditional use of the logger, but I appreciate any input on this.
Looking at the subprocess and multiprocess documentation, I think one of those might work as well but am not sure.
I'm struggling with a issue for some time now.
I'm building a little script which uses a main loop. This is a process that needs some attention from the users. The user responds on the steps and than some magic happens with use of some functions
Beside this I want to spawn another process which monitors the computer system for some specific events like pressing specif keys. If these events occur then it will launch the same functions as when the user gives in the right values.
So I need to make two processes:
-The main loop (which allows user interaction)
-The background "event scanner", which searches for specific events and then reacts on it.
I try this by launching a main loop and a daemon multiprocessing process. The problem is that when I launch the background process it starts, but after that I does not launch the main loop.
I simplified everything a little to make it more clear:
import multiprocessing, sys, time
def main_loop():
while 1:
input = input('What kind of food do you like?')
print(input)
def test():
while 1:
time.sleep(1)
print('this should run in the background')
if __name__ == '__main__':
try:
print('hello!')
mProcess = multiprocessing.Process(target=test())
mProcess.daemon = True
mProcess.start()
#after starting main loop does not start while it prints out the test loop fine.
main_loop()
except:
sys.exit(0)
You should do
mProcess = multiprocessing.Process(target=test)
instead of
mProcess = multiprocessing.Process(target=test())
Your code actually calls test in the parent process, and that call never returns.
You can use the locking synchronization to have a better control over your program's flow. Curiously, the input function raise an EOF error, but I'm sure you can find a workaround.
import multiprocessing, sys, time
def main_loop(l):
time.sleep(4)
l.acquire()
# raise an EOFError, I don't know why .
#_input = input('What kind of food do you like?')
print(" raw input at 4 sec ")
l.release()
return
def test(l):
i=0
while i<8:
time.sleep(1)
l.acquire()
print('this should run in the background : ', i+1, 'sec')
l.release()
i+=1
return
if __name__ == '__main__':
lock = multiprocessing.Lock()
#try:
print('hello!')
mProcess = multiprocessing.Process(target=test, args = (lock, ) ).start()
inputProcess = multiprocessing.Process(target=main_loop, args = (lock,)).start()
#except:
#sys.exit(0)