Develop Reference

Python multiprocessing queue not showing results

I am trying to run a separate Python Process and store the result in the queue. I can extract the result in two ways: either run queue.get() just once or use a while loop and iterate over queue until it`s empty.
In the code below first method is used if first=True and second method is used if first=False.
from multiprocessing import Process, Queue
def foo1(queue):
queue.put(1)
def main(first=False):
queue = Queue()
p = Process(target=foo1, args=(queue,))
p.start()
if first:
a = queue.get()
print(a)
else:
while not queue.empty():
print(queue.get())
p.join()
if __name__ == "__main__":
main()
Question: Why does first method print 1 correctly and second does not ? Aren`t they supposed to be equal ?
I am using Windows 10. I noticed this behavior in both interactive console and shell terminal.
Note: Due to the bug mentioned here I have to run the code as one script.

Process starts after I call a function even though I try to start the process first

I'm trying to learn how to use multiple processes in Python and I encountered a problem similar to the example below.
I try to start a process called p1 using .start() and after that to call a function do_something(). The problem is that the function is called before the process starts.
The code I used:
import time
from multiprocessing import Process
def complex_calculation():
start_timer = time.time()
print("Started calculating...")
[x ** 2 for x in range(20000000)] # calculation
print(f"complex_calculation: {time.time() - start_timer}")
def do_something():
print(input("Enter a letter: "))
if __name__ == "__main__":
p1 = Process(target=complex_calculation)
p1.start()
do_something()
p1.join()
It seems to work if I use time.sleep():
if __name__ == "__main__":
p1 = Process(target=complex_calculation)
p1.start()
time.sleep(1)
do_something()
p1.join()
My questions are:
Why does this happen?
What can I do so that I don't have to use time.sleep() ?

As pointed out in the comments, multiple processes run concurrently. Without doing some extra work, there are never guarantees about the order in which the processes are scheduled to run by the operating system. So while you call p1.start() before do_something(), all that means is that the Python code related to starting the process has completed before do_something is run. But the actual process represented by p1 may run in any way relative to the remainder of the Python code. It can run entirely before, entirely after, or interleaved in any way with the remainder of the Python code. Relying on it being scheduled in any particular way is one definition of a race condition.
To control the way in which these processes run relative to one another, you need a synchronization primitive. There are many ways to synchronize processes, it just depends on what you want to accomplish. If you want to make sure that the complex_calculation function has started before do_something is called, an event is probably the simplest approach. For example:
import time
from multiprocessing import Process, Event
def complex_calculation(event):
event.set() # Set the event, notifying any process waiting on it
start_timer = time.time()
print("Started calculating...")
[x ** 2 for x in range(20000000)] # calculation
print(f"complex_calculation: {time.time() - start_timer}")
def do_something(event):
event.wait() # Wait for `complex_calculation` to set the event
print(input("Enter a letter: "))
if __name__ == "__main__":
event = Event()
p1 = Process(target=complex_calculation, args=(event,))
p1.start()
do_something(event)
p1.join()
You should see something like:
$ python3 test.py
Started calculating...
Enter a letter: a
a
complex_calculation: 6.86732816696167

Run one Python code in the background from another Python code

I have two Python files, file 1 and file 2 that does two separate things. I want to run them together. I am using VS2017
The pseudo code for file 1 is:
Class A:
foo1():
.
.
foo2();
if variable<30;
#do this
else;
subprocess.Popen('py file2.py')
#rest of the code for foo2()
if __name__ == "__main__":
A.foo2();
Currently when I use this format, the subprocess does start the file 2 and run it but the rest of the code for foo2() after the if-else condition runs only when the process is terminated( another condition that I have setup inside file 2).
I am trying to work it in such a way that, file 2 will start running in the background once the if-else condition is met, and will give outputs in the command window but also run the rest of file 1. Not pausing the running of file 1 till file2 is done. If not in subprocess is there another way to start both files simultaneous but control the output of file 2 by passing the value of the "variable". I am trying to figure a proper work-around.
I am new to Python.
EDIT 1:
I used the command:
process = subprocess.Popen('py file2.py' ,shell=True,stdin=None, stdout=None, stderr=None, close_fds=True)
Even if I use process.kill(), the subprocess still runs in the background. It won't quit even if use the task manager.
I also wanted to pass a variable to the second file. I am looking into something like
variable = input("enter variable)
subprocess.Popen('py file2.py -a' + variable ,shell=True,stdin=None, stdout=None, stderr=None, close_fds=True)
But as far as I have looked, it was told that I can only pass strings through a subprocess. is it true?

I believe you can do this with both multithreading and multiprocessing. If you want to start them both right away and then monitor the variable, you can connect them with a pipe or queue.
starting when triggered:
from py_file2.py import your_func
import threading
Class A:
foo1():
.
.
foo2();
if variable<30;
#do this
else;
#put something here to make sure it only starts once
t = threading.Thread(target = your_func)
t.start()
#rest of the code for foo2()
if __name__ == "__main__":
A.foo2();
starting right away:
from py_file2.py import your_func
import threading
from queue import Queue
Class A:
foo1():
.
.
foo2(your_queue);
if variable<30;
#do this
else;
your_queue.put(variable)
#rest of the code for foo2()
if __name__ == "__main__":
your_queue = Queue()
t = threading.Thread(target = your_func, args = (your_queue,))
t.start()
A.foo2(your_queue);

How to run n processes simultaneously in Python

I am trying to execute n processes simultaneously. The example below works with 2 processes that are supplied externally.
At the moment it is all hard-coded for just these 2 processes but I would need to come up with the generic solution how to accomplish the same - i.e. run n processes at the same time.
My code is as follows:
import multiprocessing
'''
The first process: print 'aa'
The second Process: print 'BB'
'''
def TR1():
print 'aaaaaaaaa'
def TR2():
print 'BBBBBBBB'
if __name__ == '__main__':
process_1 = multiprocessing.Process(name='process_1', target=TR1)
process_2 = multiprocessing.Process(name='process_2', target=TR2)
process_1.start()
process_2.start()
Thanks for your suggestions!

You can either spawn processes in a loop, or use executor pool.
In real life, later one is often preferred approach, as you can limit pool size and have easy result gathering.
If you're using python 2, there's backport including ProcessPoolExecutor

How to stop multiprocessing in python running for the full script

I have the following code in python:
import multiprocessing
import time
print "I want this to show once at the beggining"
def leaveout():
print "I dont Want This to Show"
def hang(input1):
print "I want this to show 3 times"
print "Number = %s" % input1
def main():
p = multiprocessing.Process(target=hang, args=("0"))
p.start()
p1 = multiprocessing.Process(target=hang, args=("1"))
p1.start()
p2 = multiprocessing.Process(target=hang, args=("2"))
p2.start()
p.join()
p1.join()
p2.join()
if __name__ == '__main__':
main()
print "I want this to show once at the end"
My objective is to multiprocesses the hang function in three instances which is happening successfully. My issue is that the main function also runs three instances of the entire script resuting in the following output:
c:\Evopt>python multiprocessingprac.py
I want this to show once at the beggining
I want this to show once at the beggining
I want this to show once at the end
I want this to show 3 times
Number = 2
I want this to show once at the beggining
I want this to show once at the end
I want this to show 3 times
Number = 1
I want this to show once at the beggining
I want this to show once at the end
I want this to show 3 times
Number = 0
I want this to show once at the end
How can I stop this happening?

When spawning a new process, Windows creates a blank process. A new Python interpreter is then loaded in the spawned process and it's given the same code base to interpret.
That's why you see duplicated print statements being executed. As they are top level expressions, they will be executed every time a process will evaluate that code.
In Unix OSes this is not observed because it implements a totally different process creation mechanism (the fork strategy) which does not require a new Python interpreter to be loaded again in the child process.
To fix your issue, you need to remove the print( ... ) expressions from the script and move them into the main function.
def main():
print("I want this to show once at the beggining")
p0 = multiprocessing.Process( ... )
p0.start()
...
p2.join()
print("I want this to show once at the end")
You can read more about process start strategies in the multiprocessing documentation.