Develop Reference

Python concurrent futures ProcessPoolExecutor and global variables: works on Linux, error on MacOS

The code example below runs as I thought it should on two Linux machines: using Python 3.6.8 on a large CentOS-based server running Red Hat 4.8.5-39 kernel, and using Python 3.7.3 on my MX-based box running Debian 8.3.0-6 kernel).
$ python3 testshared.py filename.dat
filename.dat
270623586670000.0
However, on my Mac running Mojave 10.14.6, using Python 3.8.3, I get an error because foo=[] in function processBigFatRow(). Note that foo is assigned in getBigFatData() before starting the process pool. It's like in Linux, the version of foo assigned in getBigFatData() is passed to the processes while on Mac, the processes just uses the initialization at the top of the code (which I have to put there so they are global variable).
I understand that process are "independent copies" of the main process and that you can't assign global variables in one process and expect them to change in the other. But what about variables already set before parallel processes are started, and that are only used by reference? It's like process copies are not the same across OSs. Which one is working "as-designed"?
Code example:
import pylab as pl
from concurrent import futures
import sys
foo = []
bar = []
def getBigFatData(filename):
global foo, bar
# get the big fat data
print(filename)
foo = pl.arange(1000000).reshape(1000,1000)
# compute something as a result
bar = pl.sum(foo, axis=1)
def processBigFatRow(row):
total = pl.sum(foo[row,:]**2) if row % 5 else bar[row]
return total
def main():
getBigFatData(sys.argv[1])
grandTotal = 0.
rows = pl.arange(100)
with futures.ProcessPoolExecutor() as pool:
for tot in pool.map(processBigFatRow, rows):
grandTotal+=tot
print(grandTotal)
if __name__ == '__main__':
main()
EDIT:
As suggested, I tested Python 3.8.6 on my MX-Linux box, and it works.
So it works on Linux using Python 3.6.8, 3.7.3 and 3.8.6.
But it doesn't on Mac using Python 3.8.3.
EDIT 2:
From multiprocessing doc:
On Unix a child process can make use of a shared resource created in a parent process using a global resource.
So it won't work on Windows (and it's not the best practice), but shouldn't it work on Mac?

That is because, on MacOS, the default multiprocessing start method has changed in Python 3.8. It went from from fork (py37) to spawn (py38), causing quite its share of gnashing of teeth.
Changed in version 3.8: On macOS, the spawn start method is now the
default. The fork start method should be considered unsafe as it can
lead to crashes of the subprocess. See
bpo-33725.
With spawn: globals are not shared with multiprocess processes.
So, practically, as a quick fix, specify a 'fork' context in all of your invocations of ProcessPoolExecutor, by using mp.get_context('fork'). But be aware of the warning above; a longer-term solution would be to share variables by using one of the techniques listed on the multiprocessing docs.
For example, in your code above, replace:
with ProcessPoolExecutor() as pool:
...
with:
import multiprocessing as mp
with ProcessPoolExecutor(mp_context=mp.get_context('fork')) as executor:
...
Alternative:
When you are just writing a small script or two, and are sure that no one using a different main somewhere is going to call your code, then you can set the default start method once and for all in your main codeblock with mp.set_start_method:
if __name__ == '__main__':
mp.set_start_method('fork')
...
But generally, I prefer the first approach, as you don't have to assume that the caller has set the start method beforehand. And, as per the docs:
Note that this should be called at most once, and it should be
protected inside the if __name__ == '__main__' clause of the main
module.

You are comparing the output of the same code across two different python versions. The builtin modules could be the same, or they could have changed significantly between 3.6 and 3.8. You should run the code on the same python version in both places before going any further.

What is the correct way (if any) to use Python 2 and 3 libraries in the same program?

I wish to write a python script for that needs to do task 'A' and task 'B'. Luckily there are existing Python modules for both tasks, but unfortunately the library that can do task 'A' is Python 2 only, and the library that can do task 'B' is Python 3 only.
In my case the libraries are small and permissively-licensed enough that I could probably convert them both to Python 3 without much difficulty. But I'm wondering what is the "right" thing to do in this situation - is there some special way in which a module written in Python 2 can be imported directly into a Python 3 program, for example?

The "right" way is to translate the Py2-only module to Py3 and offer the translation upstream with a pull request (or equivalent approach for non-git upstream repos). Seriously. Horrible hacks to make py2 and py3 packages work together are not worth the effort.

I presume you know of tools such as 2to3, that aim to make the job of porting code to py3k easier, just repeating it here for others' reference.
In situations where I have to use libraries from python3 and python2, I've been able to work around it using the subprocess module. Alternatively, I've gotten around this issue with shell scripts that pipes output from the python2 script to the python3 script and vice-versa. This of course covers only a tiny fraction of use cases, but if you're transferring text (or maybe even picklable objects) between 2 & 3, it (or a more thought out variant) should work.
To the best of my knowledge, there isn't a best practice when it comes to mixing versions of python.
I present to you an ugly hack
Consider the following simple toy example, involving three files:
# py2.py
# file uses python2, here illustrated by the print statement
def hello_world():
print 'hello world'
if __name__ == '__main__':
hello_world()
# py3.py
# there's nothing py3 about this, but lets assume that there is,
# and that this is a library that will work only on python3
def count_words(phrase):
return len(phrase.split())
# controller.py
# main script that coordinates the work, written in python3
# calls the python2 library through subprocess module
# the limitation here is that every function needed has to have a script
# associated with it that accepts command line arguments.
import subprocess
import py3
if __name__ == '__main__':
phrase = subprocess.check_output('python py2.py', shell=True)
num_words = py3.count_words(phrase)
print(num_words)
# If I run the following in bash, it outputs `2`
hals-halbook: toy hal$ python3 controller.py
2

Calling a subprocess within a script using mpi4py

I’m having trouble calling an external program from my python script in which I want to use mpi4py to distribute the workload among different processors.
Basically, I want to use my script such that each core prepares some input files for calculations in separate folders, then starts an external program in this folder, waits for the output, and then, finally, reads the results and collects them.
However, I simply cannot get the external program call to work. On my search for a solution to this problem I've found that the problems I'm facing seem to be quite fundamental. The following simple example makes this clear:
#!/usr/bin/env python
import subprocess
subprocess.call(“EXTERNAL_PROGRAM”, shell=True)
subprocess.call(“echo test”, shell=True)
./script.py works fine (both calls work), while mpirun -np 1 ./script.py only outputs test. Is there any workaround for this situation? The program is definitely in my PATH, but it also fails if I use the abolute path for the call.
This SO question seems to be related, sadly there are no answers...
EDIT:
In the original version of my question I’ve not included any code using mpi4py, even though I mention this module in the title. So here is a more elaborate example of the code:
#!/usr/bin/env python
import os
import subprocess
from mpi4py import MPI
def worker(parameter=None):
"""Make new folder, cd into it, prepare the config files and execute the
external program."""
cwd = os.getcwd()
dir = "_calculation_" + parameter
dir = os.path.join(cwd, dir)
os.makedirs(dir)
os.chdir(dir)
# Write input for simulation & execute
subprocess.call("echo {} > input.cfg".format(parameter), shell=True)
subprocess.call("EXTERNAL_PROGRAM", shell=True)
# After the program is finished, do something here with the output files
# and return the data. I'm using the input parameter as a dummy variable
# for the processed output.
data = parameter
os.chdir(cwd)
return data
def run_parallel():
"""Iterate over job_args in parallel."""
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
if rank == 0:
# Here should normally be a list with many more entries, subdivided
# among all the available cores. I'll keep it simple here, so one has
# to run this script with mpirun -np 2 ./script.py
job_args = ["a", "b"]
else:
job_args = None
job_arg = comm.scatter(job_args, root=0)
res = worker(parameter=job_arg)
results = comm.gather(res, root=0)
print res
print results
if __name__ == '__main__':
run_parallel()
Unfortunately I cannot provide more details of the external executable EXTERNAL_PROGRAM other than that it is a C++ application which is MPI enabled. As written in the comment section below, I suspect that this is the reason (or one of the resons) why my external program call is basically ignored.
Please note that I’m aware of the fact that in this situation, nobody can reproduce my exact situation. Still, however, I was hoping that someone here already ran into similar problems and might be able to help.
For completeness, the OS is Ubuntu 14.04 and I’m using OpenMPI 1.6.5.

In your first example you might be able to do this:
#!/usr/bin/env python
import subprocess
subprocess.call(“EXTERNAL_PROGRAM && echo test”, shell=True)
The python script is only facilitating the MPI call. You could just as well write a bash script with command “EXTERNAL_PROGRAM && echo test” and mpirun the bash script; it would be equivalent to mpirunning the python script.
The second example will not work if EXTERNAL_PROGRAM is MPI enabled. When using mpi4py it will initialize the MPI. You cannot spawn another MPI program once you initialized the MPI environment in such a manner. You could spawn using MPI_Comm_spawn or MPI_Comm_spawn_multiple and -up option to mpirun. For mpi4py refer to Compute PI example for spawning (use MPI.COMM_SELF.Spawn).

Passing an array to a slave python script

I am quite new to python.
I learned how to pass arguments as string or floats to a slave script.
As an instance, here it is the main script:
#main script (mainscript.py)
import subprocess, sys
import numpy as np
x = np.linspace(0.5,3.2,10)
for i in range(x.size) :
subprocess.call([sys.executable,'slavescript.py',
'%s' %sys.argv[1], '%s' %sys.argv[2], '%s' %xpnt[i]])
And here the slave script:
#slave script (slavescript.py)
import sys
sys.argv[1] = str(sys.argv[1])
sys.argv[2] = int(sys.argv[2])
sys.argv[3] = float(sys.argv[3])
...
...
Now, if in python I run the following command:
run mainscript.py N 5
Then slavescript.py starts using N as a string, 5 as an integer and the third argument is converted to a float. slavescript.py is run m times, where m is the size of the array x.
I would like to pass the whole content of the array x at once, i.e. without the for loop in the main script. I think that the subprocess.call can have only strings among its arguments... I hope someone may have time to help me or give me some hints.
Thanks for the attention.
Noctu

The only reason to use a separate process is if you need parallel processing. If you do need that, then if you're managing lots of workers, use something like celery.
If you do find it appropriate to roll your own, you need to reduce what you want to send to a textual representation. I suggest using the json module.
If you don't need a separate process, just import the other python module, and access its functionality directly in code (it should already by wrapped up in functions).

How to execute an arbitrary shell script and pass multiple variables via Python?

I am building an application plugin in Python which allows users to arbitrarily extend the application with simple scripts (working under Mac OS X). Executing Python scripts is easy, but some users are more comfortable with languages like Ruby.
From what I've read, I can easily execute Ruby scripts (or other arbitrary shell scripts) using subprocess and capture their output with a pipe; that's not a problem, and there's lots of examples online. However, I need to provide the script with multiple variables (say a chunk of text along with some simple boolean information about the text the script is modifying) and I'm having trouble figuring out the best way to do this.
Does anyone have a suggestion for the best way to accomplish this? My goal is to provide scripts with the information they need with the least required code needed for accessing that information within the script.
Thanks in advance!

See http://docs.python.org/library/subprocess.html#using-the-subprocess-module
args should be a string, or a sequence
of program arguments. The program to
execute is normally the first item in
the args sequence or the string if a
string is given, but can be explicitly
set by using the executable argument.
So, your call can look like this
p = subprocess.Popen( args=["script.sh", "-p", p_opt, "-v", v_opt, arg1, arg2] )
You've put arbitrary Python values into the args of subprocess.Popen.

If you are going to be launching multiple scripts and need to pass the same information to each of them, you might consider using the environment (warning, I don't know Python, so the following code most likely sucks):
#!/usr/bin/python
import os
try:
#if environment is set
if os.environ["child"] == "1":
print os.environ["string"]
except:
#set environment
os.environ["child"] = "1"
os.environ["string"] = "hello world"
#run this program 5 times as a child process
for n in range(1, 5):
os.system(__file__)

One approach you could take would be to use json as a protocol between parent and child scripts, since json support is readily available in many languages, and is fairly expressive. You could also use a pipe to send an arbitrary amount of data down to the child process, assuming your requirements allow you to have the child scripts read from standard input. For example, the parent could do something like (Python 2.6 shown):
#!/usr/bin/env python
import json
import subprocess
data_for_child = {
'text' : 'Twas brillig...',
'flag1' : False,
'flag2' : True
}
child = subprocess.Popen(["./childscript"], stdin=subprocess.PIPE)
json.dump(data_for_child, child.stdin)
And here is a sketch of a child script:
#!/usr/bin/env python
# Imagine this were written in a different language.
import json
import sys
d = json.load(sys.stdin)
print d
In this trivial example, the output is:
$ ./foo12.py
{u'text': u'Twas brillig...', u'flag2': True, u'flag1': False}