Develop Reference

How can you create an os.environ object with a modified environment, e.g. after loading many different modules with "module load"?

I have a python script that calls an application using subprocess. I am calling this application many times, currently I am doing something along the lines of
out, err = subprocess.Popen(f"module load {' '.join(my_module_list)} && ./my_binary", stdout=sp.PIPE, stderr=sp.STDOUT, shell = True).communicate()
to run my program. Ideally I would like to first generate a modified os.environ object that already contains all the paths to the modules I am loading, and then pass it to subprocess.Popen under the env argument. However, since the printenv command doesn't output a python dictionary format, I'm not sure how to access all the modifications that modules load makes to the environment variables. Is there a good, clean way to create the required modified os.environ object?

I'd be tempted to call python in the subprocess and dump from os.environ in it
python -c 'import os; print(os.environ)'
Once you know what you're after, you can pass a dict directly to subprocess's env arg to set custom environmental vars, which could be something like
custom_env = os.environ.copy()
custom_env["foo"] = "bar"
subprocess.Popen(
...
env=custom_env,
)

PYTHONPATH variable missing when using os.execlpe to restart script as root

My end goal is to have a script that can be initially launched by a non-privileged user without using sudo, but will prompt for sudo password and self-elevate to root. I've been doing this with a bash wrapper script but would like something tidier that doesn't need an additional file.
Some googling found this question on StackOverflow where the accepted answer suggesting using os.execlpe to re-launch the script while retaining the same environment. I tried it, but it immediately failed to import a non-built-in module on the second run.
Investigating revealed that the PYTHONPATH variable is not carried over, while almost every other environment variable is (PERL5LIB is also missing, and a couple of others, but I'm not using them so they're not troubling me).
I have a brief little test script that demonstrates the issue:
#!/usr/bin/env python
import os
import sys
print(len(os.environ['PYTHONPATH']))
euid = os.geteuid()
if euid != 0:
print("Script not started as root. Running with sudo.")
args = ['sudo', sys,executable] + sys.argv + [os.environ]
os.execlpe('sudo', *args)
print("Success")
Expected output would be:
6548
Script not started as root. Running with sudo.
[sudo] password for esker:
6548
Success
But instead I'm getting a KeyError:
6548
Script not started as root. Running with sudo.
[sudo] password for esker:
Traceback (most recent call last):
File "/usr/home/esker/execlpe_test.py", line 5, in <module>
print(len(os.environ['PYTHONPATH']))
File "/vol/apps/python/2.7.6/lib/python2.7/UserDict.py", line 23, in __getitem__
raise KeyError(key)
KeyError: 'PYTHONPATH'
What would be the cause of this missing variable, and how can I avoid it disappearing? Alternatively, is there a better way about doing this that won't result in running into the problem?

I found this very weird too, and couldn't find any direct way to pass the environment into the replaced process. But I didn't do a full system debugging either.
What I found to work as a workaround is this:
pypath = os.environ.get('PYTHONPATH', "")
args = ['sudo', f"PYTHONPATH={pypath}", sys.executable] + sys.argv
os.execvpe('sudo', args, os.environ)
I.e. explicitly pass PYTHONPATH= to the new process. Note that I prefer to use os.execvpe(), but it works the same with the other exec*(), given the correct call. See this answer for a good overview of the schema.
However, PATH and the rest of the environment is still it's own environment, as an initial print(os.environ) shows. But PYTHONPATH will be passed on this way.

You're passing the environment as arguments to your script instead of arguments to execlpe. Try this instead:
args = ['sudo', sys,executable] + sys.argv + [os.environ]
os.execvpe('sudo', args, os.environ)
If you just want to inherit the environment you can even
os.execvp('sudo', args)

How do I configure PyCharm's Coverage checker to recognize .coveragerc?

I have a .coveragerc file in the root of my project. It tells coverage.py to omit my project's migrations directories:
[run]
omit = *migrations*
When I run coverage.py at the command line, the config I put in .coveragerc is obeyed.
However, PyCharm does not recognize it. Is there a setting that I'm missing?
If it turns out there's no way for PyCharm to recognize .coveragerc, I'd be happy with even just a way to omit those directories.

You can make PyCharm use the .coveragerc by putting it into the working directory you run your tests from.
The feature request from https://youtrack.jetbrains.com/issue/PY-16945 was implemented and works in version 2018.1.

There is a feature request for this at https://youtrack.jetbrains.com/issue/PY-16945

There is a different way to make PyCharm ignore certain files and folders:
In the Settings choose Project: ... - Project Structure. Here you can mark folders as Excluded or exclude files specifically.
PyCharm`s code coverage reports ignores all those exluded files, too.

I found myself in the situation where I needed this badly. My travis run were running correctly and so did coveralls but I was unable to get things work in PyCharm.
The thing is a bit hacky, but hopefully it will help people :
In my root project directory, I got a .coveragerc
[run]
omit = ./venv
concurrency = multiprocessing
parallel = True
source = HookTest
data_file = .cache/.coverage
And I "hacked" a little run_coverage.py of PyCharm : (pycharm-2016.3.2/helpers/coverage_runner/run_coverage.py )
Starting at
argv = []
Replace everything with :
argv = []
for arg in sys.argv:
if arg.startswith('-m') and arg[2:]:
argv.append(arg[2:])
else:
argv.append(arg)
cwd = os.getcwd()
rcfile = cwd + "/.coveragerc"
if os.path.exists(rcfile):
print("Loading rcfile")
i = argv.index("run")+1
argv = argv[:i] + ["--rcfile={}".format(rcfile)] + argv[i:]
sys.argv = argv
try:
main()
finally:
if run_cov:
os.chdir(cwd)
if os.getenv('COVERAGE_COMBINE'):
main(["combine"])
main(["xml", "-o", coverage_file + ".xml", "--ignore-errors"])
To make this run with python setup.py test, I created a script in PyCharm that uses said setup.py, had test has parameter, and COVERAGE_COMBINE as global env. It might not be the best of all time solutions but at least it allows me not to use HTML output anymore while working locally :)

It does not work in the last Pycharm version 2018.3.4.
The only way I succeeded to make it works is to heck the run_coverage.py as #PonteIneptique
Here is the only modification I had to do:
run_xml = os.getenv('PYCHARM_RUN_COVERAGE_XML')
argv = ["xml", "-o", coverage_file + ".xml", "--ignore-errors"]
rcfile = cwd + "/.coveragerc"
if os.path.exists(rcfile):
print("Loading rcfile\n")
argv += ["--rcfile", rcfile]
if run_xml:
os.chdir(cwd)
main(argv)
else:
try:
main()
finally:
if run_cov:
os.chdir(cwd)
main(argv)
Be sure to set the working dir of of the .coveragerc file too in your configuration.
Pycharm coders should update their code for supporting this .coveragerc file from the GUI.

Are there any modules to read shell scripts?

I'm making a python script right now, and I need to use some environment variables which are set in a bash shell script.
The bash script is something like:
#! /bin/sh
#sets some names:
export DISTRO="unified"
#export DISTRO="other"
#number of parallel builds
export BB_NUM_THREADS=2
#set build dir
export BUILDDIR=$PWD
Normally, I would just source this script in bash, then go do my builds. I'm trying to wrap python around the whole process to do some management of the output so I want to remove the manual source ./this_script.sh step.
What I want to do is read this script from python and then use os.environ to set up the variables within it. (I know this will not affect the parent, but only the current running Python instance and that's fine)
So to make my work easier, I'm trying to find out are there any modules which can "parse" the bash script and make use of the environment variables found within? Currently I'm doing this by hand and it's a bit of a pain.
If no such module exists to do exactly what I want, is there a more pythonic (read: easier/shorter) way of manually parsing a file in general, right now I'm doing this:
def parse_bash_script(fn):
with open(fn) as f:
for line in f:
if not line[:1] == '#': #ignore comments
if "export" in line:
line = line.replace(" ","").strip()
var = line[6:line.find("=")]
val = line[line.find("=")+1:len(line)]
if "\"" in val:
val = val[1:-1]
os.environ[var]=val

There is no module to do exactly what you want, but shlex will do a lot of what you want. In particular, it will get the quoting, etc. right without you having to worry about it (which is the hardest part of this), as well as skipping comments, etc. The only thing it won't do is handle the export keywords.
The easy way around that is to preprocess:
with open(fn) as f:
processed = f.read().replace('export ', '')
for line in shlex.split(processed):
var, _, value = line.partition('=')
os.environ[var] = val
It's a bit hackier, but you can also do it a bit less verbosely by post-processing. In particular, shlex will treat export foo="bar spam eggs" as two values: export and foo="bar spam eggs", and you can just skip the ones that == 'export', or where the partition finds nothing, or… For example:
with open(fn) as f:
for line in shlex.split(f.read()):
var, eq, value = line.partition('=')
if eq:
os.environ[var] = val
If you want to get fancier, you can construct a shlex object and (a) drive the parser directly from the file, and (b) control the parsing at a finer-grained level. However, I don't think that's necessary here.
Meanwhile, if you want to handle environment substitution (as the BUILDDIR=$PWD implies), this won't magically take care of that for you. You can make configparser do that for you with its ExtendedInterpolation feature, but then you'll need to trick configparser into handling shlex syntax, at which point… why bother.
You can of course do it manually by writing your own interpolator, but that's hard to get right. You need to know the shell's rules for why $PWD-foo is the same as ${PWD}-foo, but $PWD_foo is the same as ${PWD_foo}, etc.
A better solution at this point—assuming the script is actually safe to run—would be to actually use a shell to do it for you. For example:
with open('script.sh') as f:
script = f.read()
script += b'\nenv'
with subprocess.Popen(['sh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE) as p:
result = p.communicate(script)
for line in result.splitlines():
var, _, value = line.partition('=')
os.environ[var] = value
Of course this will also override things like _=/usr/bin/env, but probably not anything you care about.

def parse_bash_script(fn):
with open(fn) as f:
for line in f:
if not line.startswith('#'): #ignore comments
if "export" in line:
var, _, val = line.partition('=')
var = var.lstrip()
val = val.rstrip()
if val.startswith('"'):
vals = val.rpartition('"')
val = vals[0][1]+vals[2]
os.environ[var]=val

I had the same problem, and based on the advice from abarnert, I decided to implement the solution as a subprocess call to a restricted bash shell, combined with shlex.
import shlex
import subprocess
filename = '/path/to/file.conf'
o, e = subprocess.Popen(
['/bin/bash', '--restricted', '--noprofile', '--init-file',
filename, '-i', '-c', 'declare'],
env={'PATH': ''},
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()
if e:
raise StandardError('conf error in {}: {}'.format(filename, e))
for token in shlex.split(o):
parts = token.split('=', 1)
if len(parts) == 2:
os.environ[parts[0]] = parts[1]
The advantage to the restricted shell is that it blocks many of the undesirable or malicious side effects that may otherwise happen when executing a shell script. From the bash documentation:
A restricted shell is used to set up an environment more controlled than the standard shell. It behaves identically to bash with the exception that the following are disallowed or not performed:
changing directories with cd
setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV
specifying command names containing /
specifying a file name containing a / as an argument to the . builtin command
Specifying a filename containing a slash as an argument to the -p option to the hash builtin command
importing function definitions from the shell environment at startup
parsing the value of SHELLOPTS from the shell environment at startup
redirecting output using the >, >|, <>, >&, &>, and >> redirection operators
using the exec builtin command to replace the shell with another command
adding or deleting builtin commands with the -f and -d options to the enable builtin command
Using the enable builtin command to enable disabled shell builtins
specifying the -p option to the command builtin command
turning off restricted mode with set +r or set +o restricted.

Best way to specify path to binary for a wrapper module

I write a module that wraps functionality of an external binary.
For example, I wrap ls program into a python module my_wrapper.py
import my_wrapper
print my_wrapper.ls('some_directory/')
# list files in some_directory
and in my_wrapper.py I do:
# my_wrapper.py
PATH_TO_LS = '/bin/ls'
def ls(path):
proc = subprocess.Popen([PATH_TO_LS, path], ...)
...
return paths
(of course, I do not wrap ls but some other binary)
The binary might be installed with an arbitrary location, like /usr/bin/, /opt/ or even at the same place as the python script (./binaries/)
Question:
What would be the cleanest (from the user perspective) way to set the path to the binary?
Should the user specify my_wrapper.PATH_TO_LS = ... or invoke some my_wrapper.set_binary_path(path) at the beginning of his script?
Maybe it would be better to specify it in env, and the wrapper would find it with os.environ?
If the wrapper is distributed as egg, can I require during the installation, that the executable is already present in the system (see below)?
egg example:
# setup.py
setup(
name='my_wrapper',
requires_binaries=['the_binary'] # <--- require that the binary is already
# installed and on visible
# on execution path
)
or
easy_install my_wrapper BINARY_PATH=/usr/local/bin/the_binary

Create a "configuration object" with sane defaults. Allow the consumer to modify the values as appropriate. Accept a configuration object instance to your functions, taking the one you created by default.