Develop Reference

How can you create an os.environ object with a modified environment, e.g. after loading many different modules with "module load"?

I have a python script that calls an application using subprocess. I am calling this application many times, currently I am doing something along the lines of
out, err = subprocess.Popen(f"module load {' '.join(my_module_list)} && ./my_binary", stdout=sp.PIPE, stderr=sp.STDOUT, shell = True).communicate()
to run my program. Ideally I would like to first generate a modified os.environ object that already contains all the paths to the modules I am loading, and then pass it to subprocess.Popen under the env argument. However, since the printenv command doesn't output a python dictionary format, I'm not sure how to access all the modifications that modules load makes to the environment variables. Is there a good, clean way to create the required modified os.environ object?

I'd be tempted to call python in the subprocess and dump from os.environ in it
python -c 'import os; print(os.environ)'
Once you know what you're after, you can pass a dict directly to subprocess's env arg to set custom environmental vars, which could be something like
custom_env = os.environ.copy()
custom_env["foo"] = "bar"
subprocess.Popen(
...
env=custom_env,
)

Set environment variable of calling bash script in python [duplicate]

This question already has answers here:
Is it possible to change the Environment of a parent process in Python?
(4 answers)
Closed 4 years ago.
I have a bash script that looks like this:
python myPythonScript.py
python myOtherScript.py $VarFromFirstScript
and myPythonScript.py looks like this:
print("Running some code...")
VarFromFirstScript = someFunc()
print("Now I do other stuff")
The question is, how do I get the variable VarFromFirstScript back to the bash script that called myPythonScript.py.
I tried os.environ['VarFromFirstScript'] = VarFromFirstScript but this doesn't work (I assume this means that the python environment is a different env from the calling bash script).

you cannot propagate an environment variable to the parent process. But you can print the variable, and assign it back to the variable name from your shell:
VarFromFirstScript=$(python myOtherScript.py $VarFromFirstScript)
you must not print anything else in your code, or using stderr
sys.stderr.write("Running some code...\n")
VarFromFirstScript = someFunc()
sys.stdout.write(VarFromFirstScript)
an alternative would be to create a file with the variables to set, and make it parse by your shell (you could create a shell that the parent shell would source)
import shlex
with open("shell_to_source.sh","w") as f:
f.write("VarFromFirstScript={}\n".format(shlex.quote(VarFromFirstScript))
(shlex.quote allows to avoid code injection from python, courtesy Charles Duffy)
then after calling python:
source ./shell_to_source.sh

You can only pass environment variables from parent process to child.
When the child process is created the environment block is copied to the child - the child has a copy, so any changes in the child process only affects the child's copy (and any further children which it creates).
To communicate with the parent the simplest way is to use command substitution in bash where we capture stdout:
Bash script:
#!/bin/bash
var=$(python myPythonScript.py)
echo "Value in bash: $var"
Python script:
print("Hollow world!")
Sample run:
$ bash gash.sh
Value in bash: Hollow world!
You have other print statements in python, you will need to filter out to only the data you require, possibly by marking the data with a well-known prefix.
If you have many print statements in python then this solution is not scalable, so you might need to use process substitution, like this:
Bash script:
#!/bin/bash
while read -r line
do
if [[ $line = ++++* ]]
then
# Strip out the marker
var=${line#++++}
else
echo "$line"
fi
done < <(python myPythonScript.py)
echo "Value in bash: $var"
Python script:
def someFunc():
return "Hollow World"
print("Running some code...")
VarFromFirstScript = someFunc()
# Prefix our data with a well-known marker
print("++++" + VarFromFirstScript)
print("Now I do other stuff")
Sample Run:
$ bash gash.sh
Running some code...
Now I do other stuff
Value in bash: Hollow World

I would source your script, this is the most commonly used method. This executes the script under the current shell instead of loading another one. Because this uses same shell env variables you set will be accessible when it exits. . /path/to/script.sh or source /path/to/script.sh will both work, . works where source doesn't sometimes.

Import bash variables from a python script

I have seen plenty examples of running a python script from inside a bash script and either passing in variables as arguments or using export to give the child shell access, I am trying to do the opposite here though.
I am running a python script and have a separate file, lets call it myGlobalVariables.bash
myGlobalVariables.bash:
foo_1="var1"
foo_2="var2"
foo_3="var3"
My python script needs to use these variables.
For a very simple example:
myPythonScript.py:
print "foo_1: {}".format(foo_1)
Is there a way I can import them directly? Also, I do not want to alter the bash script if possible since it is a common file referenced many times elsewhere.

If your .bash file is formatted as you indicated - you might be able to just import it direct as a Python module via the imp module.
import imp
bash_module = imp.load_source("bash_module, "/path/to/myGlobalVariables.bash")
print bash_module.foo_1

You can also use os.environ:
Bash:
#!/bin/bash
# works without export as well
export testtest=one
Python:
#!/usr/bin/python
import os
os.environ['testtest'] # 'one'

I am very new to python, so I would welcome suggestions for more idiomatic ways to do this, but the following code uses bash itself to tell us which values get set by first calling bash with an empty environment (env -i bash) to tell us what variables are set as a baseline, then I call it again and tell bash to source your "variables" file, and then tell us what variables are now set. After removing some false-positives and an apparently-blank line, I loop through the "additional" output, looking for variables that were not in the baseline. Newly-seen variables get split (carefully) and put into the bash dictionary. I've left here (but commented-out) my previous idea for using exec to set the variables natively in python, but I ran into quoting/escaping issues, so I switched gears to using a dict.
If the exact call (path, etc) to your "variables" file is different than mine, then you'll need to change all of the instances of that value -- in the subprocess.check_output() call, in the list.remove() calls.
Here's the sample variable file I was using, just to demonstrate some of the things that could happen:
foo_1="var1"
foo_2="var2"
foo_3="var3"
if [[ -z $foo_3 ]]; then
foo_4="test"
else
foo_4="testing"
fi
foo_5="O'Neil"
foo_6='I love" quotes'
foo_7="embedded
newline"
... and here's the python script:
#!/usr/bin/env python
import subprocess
output = subprocess.check_output(['env', '-i', 'bash', '-c', 'set'])
baseline = output.split("\n")
output = subprocess.check_output(['env', '-i', 'bash', '-c', '. myGlobalVariables.bash; set'])
additional = output.split("\n")
# these get set when ". myGlobal..." runs and so are false positives
additional.remove("BASH_EXECUTION_STRING='. myGlobalVariables.bash; set'")
additional.remove('PIPESTATUS=([0]="0")')
additional.remove('_=myGlobalVariables.bash')
# I get an empty item at the end (blank line from subprocess?)
additional.remove('')
bash = {}
for assign in additional:
if not assign in baseline:
name, value = assign.split("=", 1)
bash[name]=value
#exec(name + '="' + value + '"')
print "New values:"
for key in bash:
print "Key: ", key, " = ", bash[key]
Another way to do it:
Inspired by Marat's answer, I came up with this two-stage hack. Start with a python program, let's call it "stage 1", which uses subprocess to call bash to source the variable file, as my above answer does, but it then tells bash to export all of the variables, and then exec the rest of your python program, which is in "stage 2".
Stage 1 python program:
#!/usr/bin/env python
import subprocess
status = subprocess.call(
['bash', '-c',
'. myGlobalVariables.bash; export $(compgen -v); exec ./stage2.py'
]);
Stage 2 python program:
#!/usr/bin/env python
# anything you want! for example,
import os
for key in os.environ:
print key, " = ", os.environ[key]

As stated in #theorifice answer, the trick here may be that such formatted file may be interpreted by both as bash and as python code. But his answer is outdated. imp module is deprecated in favour of importlib.
As your file has extension other than ".py", you can use the following approach:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("foobar", SourceFileLoader("foobar", "myGlobalVariables.bash"))
foobar = module_from_spec(spec)
spec.loader.exec_module(foobar)
I do not completely understand how this code works (where there are these foobar parameters), however, it worked for me. Found it here.

Load environment variables from a shell script

I have a file with some environment variables that I want to use in a python script
The following works form the command line
$ source myFile.sh
$ python ./myScript.py
and from inside the python script I can access the variables like
import os
os.getenv('myvariable')
How can I source the shell script, then access the variables, from with the python script?

If you are saying backward environment propagation, sorry, you can't. It's a security issue. However, directly source environment from python is definitely valid. But it's more or less a manual process.
import subprocess as sp
SOURCE = 'your_file_path'
proc = sp.Popen(['bash', '-c', 'source {} && env'.format(SOURCE)], stdout=sp.PIPE)
source_env = {tup[0].strip(): tup[1].strip() for tup in map(lambda s: s.strip().split('=', 1), proc.stdout)}
Then you have everything you need in source_env.
If you need to write it back to your local environment (which is not recommended, since source_env keeps you clean):
import os
for k, v in source_env.items():
os.environ[k] = v
Another tiny attention needs to be paid here, is since I called bash here, you should expect the rules are applied here too. So if you want your variable to be seen, you will need to export them.
export VAR1='see me'
VAR2='but not me'

You can not load environmental variables in general from a bash or shell script, it is a different language. You will have to use bash to evaluate the file and then somehow print out the variables and then read them. see Forcing bash to expand variables in a string loaded from a file

Are there any modules to read shell scripts?

I'm making a python script right now, and I need to use some environment variables which are set in a bash shell script.
The bash script is something like:
#! /bin/sh
#sets some names:
export DISTRO="unified"
#export DISTRO="other"
#number of parallel builds
export BB_NUM_THREADS=2
#set build dir
export BUILDDIR=$PWD
Normally, I would just source this script in bash, then go do my builds. I'm trying to wrap python around the whole process to do some management of the output so I want to remove the manual source ./this_script.sh step.
What I want to do is read this script from python and then use os.environ to set up the variables within it. (I know this will not affect the parent, but only the current running Python instance and that's fine)
So to make my work easier, I'm trying to find out are there any modules which can "parse" the bash script and make use of the environment variables found within? Currently I'm doing this by hand and it's a bit of a pain.
If no such module exists to do exactly what I want, is there a more pythonic (read: easier/shorter) way of manually parsing a file in general, right now I'm doing this:
def parse_bash_script(fn):
with open(fn) as f:
for line in f:
if not line[:1] == '#': #ignore comments
if "export" in line:
line = line.replace(" ","").strip()
var = line[6:line.find("=")]
val = line[line.find("=")+1:len(line)]
if "\"" in val:
val = val[1:-1]
os.environ[var]=val

There is no module to do exactly what you want, but shlex will do a lot of what you want. In particular, it will get the quoting, etc. right without you having to worry about it (which is the hardest part of this), as well as skipping comments, etc. The only thing it won't do is handle the export keywords.
The easy way around that is to preprocess:
with open(fn) as f:
processed = f.read().replace('export ', '')
for line in shlex.split(processed):
var, _, value = line.partition('=')
os.environ[var] = val
It's a bit hackier, but you can also do it a bit less verbosely by post-processing. In particular, shlex will treat export foo="bar spam eggs" as two values: export and foo="bar spam eggs", and you can just skip the ones that == 'export', or where the partition finds nothing, or… For example:
with open(fn) as f:
for line in shlex.split(f.read()):
var, eq, value = line.partition('=')
if eq:
os.environ[var] = val
If you want to get fancier, you can construct a shlex object and (a) drive the parser directly from the file, and (b) control the parsing at a finer-grained level. However, I don't think that's necessary here.
Meanwhile, if you want to handle environment substitution (as the BUILDDIR=$PWD implies), this won't magically take care of that for you. You can make configparser do that for you with its ExtendedInterpolation feature, but then you'll need to trick configparser into handling shlex syntax, at which point… why bother.
You can of course do it manually by writing your own interpolator, but that's hard to get right. You need to know the shell's rules for why $PWD-foo is the same as ${PWD}-foo, but $PWD_foo is the same as ${PWD_foo}, etc.
A better solution at this point—assuming the script is actually safe to run—would be to actually use a shell to do it for you. For example:
with open('script.sh') as f:
script = f.read()
script += b'\nenv'
with subprocess.Popen(['sh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE) as p:
result = p.communicate(script)
for line in result.splitlines():
var, _, value = line.partition('=')
os.environ[var] = value
Of course this will also override things like _=/usr/bin/env, but probably not anything you care about.

def parse_bash_script(fn):
with open(fn) as f:
for line in f:
if not line.startswith('#'): #ignore comments
if "export" in line:
var, _, val = line.partition('=')
var = var.lstrip()
val = val.rstrip()
if val.startswith('"'):
vals = val.rpartition('"')
val = vals[0][1]+vals[2]
os.environ[var]=val

I had the same problem, and based on the advice from abarnert, I decided to implement the solution as a subprocess call to a restricted bash shell, combined with shlex.
import shlex
import subprocess
filename = '/path/to/file.conf'
o, e = subprocess.Popen(
['/bin/bash', '--restricted', '--noprofile', '--init-file',
filename, '-i', '-c', 'declare'],
env={'PATH': ''},
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()
if e:
raise StandardError('conf error in {}: {}'.format(filename, e))
for token in shlex.split(o):
parts = token.split('=', 1)
if len(parts) == 2:
os.environ[parts[0]] = parts[1]
The advantage to the restricted shell is that it blocks many of the undesirable or malicious side effects that may otherwise happen when executing a shell script. From the bash documentation:
A restricted shell is used to set up an environment more controlled than the standard shell. It behaves identically to bash with the exception that the following are disallowed or not performed:
changing directories with cd
setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV
specifying command names containing /
specifying a file name containing a / as an argument to the . builtin command
Specifying a filename containing a slash as an argument to the -p option to the hash builtin command
importing function definitions from the shell environment at startup
parsing the value of SHELLOPTS from the shell environment at startup
redirecting output using the >, >|, <>, >&, &>, and >> redirection operators
using the exec builtin command to replace the shell with another command
adding or deleting builtin commands with the -f and -d options to the enable builtin command
Using the enable builtin command to enable disabled shell builtins
specifying the -p option to the command builtin command
turning off restricted mode with set +r or set +o restricted.