I was hoping to write a python script to create some appropriate environmental variables by running the script in whatever directory I'll be executing some simulation code, and I've read that I can't write a script to make these env vars persist in the mac os terminal. So two things:
Is this true?
and
It seems like it would be a useful things to do; why isn't it possible in general?
You can't do it from python, but some clever bash tricks can do something similar. The basic reasoning is this: environment variables exist in a per-process memory space. When a new process is created with fork() it inherits its parent's environment variables. When you set an environment variable in your shell (e.g. bash) like this:
export VAR="foo"
What you're doing is telling bash to set the variable VAR in its process space to "foo". When you run a program, bash uses fork() and then exec() to run the program, so anything you run from bash inherits the bash environment variables.
Now, suppose you want to create a bash command that sets some environment variable DATA with content from a file in your current directory called ".data". First, you need to have a command to get the data out of the file:
cat .data
That prints the data. Now, we want to create a bash command to set that data in an environment variable:
export DATA=`cat .data`
That command takes the contents of .data and puts it in the environment variable DATA. Now, if you put that inside an alias command, you have a bash command that sets your environment variable:
alias set-data="export DATA=`cat .data`"
You can put that alias command inside the .bashrc or .bash_profile files in your home directory to have that command available in any new bash shell you start.
One workaround is to output export commands, and have the parent shell evaluate this..
thescript.py:
import pipes
import random
r = random.randint(1,100)
print("export BLAHBLAH=%s" % (pipes.quote(str(r))))
..and the bash alias (the same can be done in most shells.. even tcsh!):
alias setblahblahenv="eval $(python thescript.py)"
Usage:
$ echo $BLAHBLAH
$ setblahblahenv
$ echo $BLAHBLAH
72
You can output any arbitrary shell code, including multiple commands like:
export BLAHBLAH=23 SECONDENVVAR='something else' && echo 'everything worked'
Just remember to be careful about escaping any dynamically created output (the pipes.quote module is good for this)
If you set environment variables within a python script (or any other script or program), it won't affect the parent shell.
Edit clarification:
So the answer to your question is yes, it is true.
You can however export from within a shell script and source it by using the dot invocation
in fooexport.sh
export FOO="bar"
at the command prompt
$ . ./fooexport.sh
$ echo $FOO
bar
It's not generally possible. The new process created for python cannot affect its parent process' environment. Neither can the parent affect the child, but the parent gets to setup the child's environment as part of new process creation.
Perhaps you can set them in .bashrc, .profile or the equivalent "runs on login" or "runs on every new terminal session" script in MacOS.
You can also have python start the simulation program with the desired environment. (use the env parameter to subprocess.Popen (http://docs.python.org/library/subprocess.html) )
import subprocess, os
os.chdir('/home/you/desired/directory')
subprocess.Popen(['desired_program_cmd', 'args', ...], env=dict(SOMEVAR='a_value') )
Or you could have python write out a shell script like this to a file with a .sh extension:
export SOMEVAR=a_value
cd /home/you/desired/directory
./desired_program_cmd
and then chmod +x it and run it from anywhere.
What I like to do is use /usr/bin/env in a shell script to "wrap" my command line when I find myself in similar situations:
#!/bin/bash
/usr/bin/env NAME1="VALUE1" NAME2="VALUE2" ${*}
So let's call this script "myappenv". I put it in my $HOME/bin directory which I have in my $PATH.
Now I can invoke any command using that environment by simply prepending "myappenv" as such:
myappenv dosometask -xyz
Other posted solutions work too, but this is my personal preference. One advantage is that the environment is transient, so if I'm working in the shell only the command I invoke is affected by the altered environment.
Modified version based on new comments
#!/bin/bash
/usr/bin/env G4WORKDIR=$PWD ${*}
You could wrap this all up in an alias too. I prefer the wrapper script approach since I tend to have other environment prep in there too, which makes it easier for me to maintain.
As answered by Benson, but the best hack-around is to create a simple bash function to preserve arguments:
upsert-env-var (){ eval $(python upsert_env_var.py $*); }
Your can do whatever you want in your python script with the arguments. To simply add a variable use something like:
var = sys.argv[1]
val = sys.argv[2]
if os.environ.get(var, None):
print "export %s=%s:%s" % (var, val, os.environ[var])
else:
print "export %s=%s" % (var, val)
Usage:
upsert-env-var VAR VAL
As others have pointed out, the reason this doesn't work is that environment variables live in a per-process memory spaces and thus die when the Python process exits.
They point out that a solution to this is to define an alias in .bashrc to do what you want such as this:
alias export_my_program="export MY_VAR=`my_program`"
However, there's another (a tad hacky) method which does not require you to modify .bachrc, nor requires you to have my_program in $PATH (or specify the full path to it in the alias). The idea is to run the program in Python if it is invoked normally (./my_program), but in Bash if it is sourced (source my_program). (Using source on a script does not spawn a new process and thus does not kill environment variables created within.) You can do that as follows:
my_program.py:
#!/usr/bin/env python3
_UNUSED_VAR=0
_UNUSED_VAR=0 \
<< _UNUSED_VAR
#=======================
# Bash code starts here
#=======================
'''
_UNUSED_VAR
export MY_VAR=`$(dirname $0)/my_program.py`
echo $MY_VAR
return
'''
#=========================
# Python code starts here
#=========================
print('Hello environment!')
Running this in Python (./my_program.py), the first 3 lines will not do anything useful and the triple-quotes will comment out the Bash code, allowing Python to run normally without any syntax errors from Bash.
Sourcing this in bash (source my_program.py), the heredoc (<< _UNUSED_VAR) is a hack used to "comment out" the first-triple quote, which would otherwise be a syntax error. The script returns before reaching the second triple-quote, avoiding another syntax error. The export assigns the result of running my_program.py in Python from the correct directory (given by $(dirname $0)) to the environment variable MY_VAR. echo $MY_VAR prints the result on the command-line.
Example usage:
$ source my_program.py
Hello environment!
$ echo $MY_VAR
Hello environment!
However, the script will still do everything it did before except exporting, the environment variable if run normally:
$ ./my_program.py
Hello environment!
$ echo $MY_VAR
<-- Empty line
As noted by other authors, the memory is thrown away when the Python process exits. But during the python process, you can edit the running environment. For example:
>>> os.environ["foo"] = "bar"
>>> import subprocess
>>> subprocess.call(["printenv", "foo"])
bar
0
>>> os.environ["foo"] = "foo"
>>> subprocess.call(["printenv", "foo"])
foo
0
Related
I would like to build a python script, which can manipulate the state of it's calling bash shell, especially it's working directory for the beginning.
With os.chdir or os.system("ls ..") you can only change the interpreters path, but how can I apply the comments changes to the scripts caller then?
Thank you for any hint!
You can't do that directly from python, as a child process can never change the environment of its parent process.
But you can create a shell script that you source from your shell, i.e. it runs in the same process, and in that script, you'll call python and use its output as the name of the directory to cd to:
/home/choroba $ cat 1.sh
cd "$(python -c 'print ".."')"
/home/choroba $ . 1.sh
/home $
When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.
I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.
As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)
If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.
To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.
It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())
If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python
/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py
Maybe you need to check the file privileges:
sudo chmod +x script.py
I'm pretty new to Python and trying to find an approach for setting up several shell env variables and then executing a shell script.
Is the subprocess module capable of sending several shell commands and capturing the output?
This isn't clear to me, since each spawn will result in a new shell thread that will not have the previous env variable.
Also, would it be appropriate to use popen or check_output?
Essentially what I'd like to do is:
$ setenv a b
$ setenv c d
$ setenv e f
$ check_status.sh > log.log
$ grep "pattern" log.log
Where check_status.sh must have the above env variables defined to run properly, and it also must be shell script (i.e. I can't translate check_status.sh to python).
I'd appreciate you comments and input.
subprocess will create one thread everytime. You can use popen to execute shell command. Use semicolons as separators. Such as os.popen('setenv a b;set env c d')
Hope it helps.
Well, I'm trying to using a python3 script to manage my aliases on my MAC OS X. At first I've put all alias commands in a single file and try to use below code to turn on/off these alias:
def enable_alias(self):
alias_controller = AliasListControl() # just a simple class to handle the single file path and other unimportant things.
os.popen('cp ~/.bash_aliases ~/.bash_aliases.bak')
os.popen('cat ' + alias_controller.path + '>> ~/.bash_aliases')
os.system('source ~/.bash_aliases')
def disable_alias(self):
os.popen('mv ~/.bash_aliases.bak ~/.bash_aliases')
os.popen('source ~/.bash_aliases')# maybe I should call some other unalias commands there
As you see, there exists an problem. When the script runs to os.system('source ~/.bash_aliases'), it will first open A subshell and execute the command, so the source operation will only take effect in the subshell, not the parent shell, then the command finished and the subshell was closed. This means what os.system('source ~/.bash_aliases') has done is just in vein.
It doesn't address your process problem, but an alternative is to put your commands either into shell scripts or into function definitions that are defined in your ~/.bash_profile.
For example, as a script:
Create the file enable_alias.sh:
filename=$1
cp ~/.bash_aliases ~/.bash_aliases.bak
# If you use `cat` here, your aliases file will keep getting longer and longer with repeated definitions... think you want to use > not >>
cp /path/to/$1.txt ~/.bash_aliases
source ~/.bash_aliases
Put this file somewhere in a folder in your PATH and make it executable. Then run it as:
enable_alias.sh A
..where your file of settings, etc is called A.txt. The $1 will pass the first value as the file name.
Or alternatively, you could do it as a function, and add that definition to your .bash_profile. (Functions can also take $1 when called.)
disable_alias(){
mv ~/.bash_aliases.bak ~/.bash_aliases
source ~/.bash_aliases
}
As you say, it might be a good idea to put unalias commands into your .bash_aliases file as well. It might also be simpler to have copies of aliases as their own files A.txt B.txt etc and just cp A.txt ~/.bash_aliases with the enable command and not use the disable command at all (disable is equivalent to enabling file B.txt, for example.)
Just some thoughts on another approach that is more 'bash-like'...
I'm not familiar with OS/X, but I am familiar with bash, so I'll take a shot at this.
First, look into Python's shutil module and/or subprocess module; os.system and os.popen are no longer the best way of doing these things.
Second, don't source a script from a subshell that's going to go away immediately afterward. Instead, add something like:
source ~/.bash_aliases
in your ~/.bashrc, so that it'll get used when every new bash is started.
I'm currently trying to write some component tests for my team using python and I ran into a test procedure that tells the tester to source a csh file. This file has a bunch of setenv commands and aliases. All of these environment variables are needed by the next executable in the chain. I wanted to devise some way to extract all of the env vars and pass them to the next process in the chain.
I looked at the following question, which is is almost what I need:
Emulating Bash 'source' in Python
It pushes all exported bash environment variables into a dictionary, which I can then pass to the next process. The problem is this seems to only work for export and not the csh setenv command.
If this isn't possible, is there a way to run the .csh file with a subprocess command such as /bin/sh -c script.csh and then run the process that needs those environment variables as a subprocess to that process (so that it could inherit it's environment variables?)
Essentially, I have a process and script that has a bunch of setenv variables and that process needs to run in an environment that contains all of those environment variables.
Thanks for any help,
Have you considered John Kugelman's solution?
That would be the simplest way.
However, if for some reason you need or want Python to be an intermediary between these scripts, then you could use the following to source the script and retrieve the environment variables. (Edit: thanks to abarnert and Keith Thompson for the correct csh command.)
If script.csh contains
setenv foo bar
then
import subprocess
import os
import json
PIPE = subprocess.PIPE
output = subprocess.check_output(
'''source script.csh; python -c "import os, json; print(json.dumps(dict(os.environ)))"''',
shell=True, executable='/bin/csh')
env = json.loads(output)
print(env['foo'])
yields
bar
The simplest way to do this is to let csh source the setup script and then run the other script. For example:
setup.csh:
setenv foo bar
script.csh:
echo $foo
runner.py:
import subprocess
csh_command = 'source ./setup.csh && ./script.csh'
subprocess.check_call(['/bin/csh', '-c', csh_command])
If you just want to use the code from the linked question, all you have to do is change the regex. Just replace export with setenv, and the = with \s+.
So script.csh contains setenv commands that set certain environment variables.
/bin/sh -c script.csh will just execute the script; the environment variables will be lost as soon as the script finishes. Even that assumes that script.csh is executable; if it's meant to be sourced, it needn't be, and probably shouldn't be.
For the environment variable settings to be visible, script.csh has to be sourced by a csh process.
Here's a rough outline:
/bin/csh -c 'source script.csh ; python -c ...'
The python -c ... process will be able to see the environment variables. The process that executes the above command will not.
A more complex solution would be something like this:
printenv > before.txt
/bin/csh -c 'source script.csh ; printenv' > after.txt
And then compare before.txt and after.txt to see what changed in the environment. (Or you can capture the output in some other way.) That has the advantage of letting the calling process see the modified environment, or at least obtain information about it.
Note that in some unusual cases, you may not be able to determine the environment from the output of printenv. For example, if the value of $FOO is "10\nBAR=20", that will appear in the output of printenv as:
FOO=10
BAR=20
which looks like $FOO is 10 and $BAR is 20. (It's actually not uncommon to have newlines in environment variable values; my $TERMCAP is currently set to a 21-line chunk of text.
You could avoid that by writing your own printenv replacement that prints an unambiguous representation of the environment.
This works for me to get the environment from a shell script (that calls other shell scripts). You could do something like this to read the env into a dictionary and then use os.environ to change the env.
import subprocess
SAPENVCMD = "/bin/csh -c \'source ~/.cshrc && env\'"
envdict = {}
def runcommand(command):
ps = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out,err = ps.communicate()
outlist = out.split('\n')
if ps.returncode == 0:
return outlist
else:
return None
def elist2dict(dict,list):
for entry in list:
if entry is not None:
pos = entry.find('=')
key = entry[:pos]
pos += 1
value = entry[pos:]
dict.setdefault(key,value)
def env2dict(dict,ENVCMD):
envlist = runcommand(ENVCMD)
elist2dict(dict,envlist)
env2dict(envdict,SAPENVCMD)
v = envdict["SAPSYSTEMNAME"]
print v