Develop Reference

Get the a variable defined in the command running python

I have this script named test.py:
import os
print(os.environ['LOL'])
That I run as follow :
(LOL=HAHAHA; python3 test.py)
But it raises a KeyError because it can't find the variable LOL.
I also tried with :
os.getenv('LOL')
But it just returns None.
How can I access to the variable LOL in this context.

You need to supplement the environment variables for the command during which you invoke python3 test.py as follows
$ LOL=HAHAHA python3 test.py
HAHAHA
$ env LOL=HAHAHA python3 test.py
HAHAHA
$ echo $LOL
<empty string>
or you can export the variable for the current session as:
$ export LOL=HAHAHA
$ python3 test.py
HAHAHA
$ echo $LOL
HAHAHA
Simply doing LOL=HAHAHA; python3 test.py doesn't work because that just sets the LOL=HAHAHA variable for the shell process.
Another thing to note, the first approach shown only sets the environment variable for that specific command. It does not set it in the environment. Doing it with the export instead, sets it for the environment. You can see the difference in the values of $LOL above

You are trying to access an environmental variable, so if you are on windows to set it you need to do something like:
set LOL=HAHAHAHA
Then you should be able to access it. To make sure it was set correctly you can also just run:
set
To get a full list of environmental variables.

Just export the variable, i.e. run the command with:
export LOL=HAHA; python3 test.py
or set LOL in the same command, i.e. without the semicolon:
LOL=HAHA python3 test.py
Content of test.py:
import os
print(os.environ['LOL'])

What shebang to use for Python scripts run under a pyenv virtualenv

When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.

I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.

As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)

If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.

To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.

It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())

If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python

/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py

Maybe you need to check the file privileges:
sudo chmod +x script.py

Python and Bash command line env vars were handled different?

$ cat test.py
#!/usr/bin/env python
# coding=utf-8
import os
print (os.environ.get('test'))
$ test=4 python test.py
4
$ test=4; python test.py
None
While in shell I got different with python:
$ test=4; echo $test
4
But :
$ test=2
$ test=4 echo $test
2
So I am confused about how python and bash handle the situation. Can someone explain ?

Thats the difference between shell and environment variable.
Here,
test=4 python test.py
passes test=4 to python's environment, so you will get the variable test inside the script.
Whereas
test=4; python test.py
creates a shell variable that is available only in the current shell session (that is why you are getting the value from shell just fine) i.e. will not be propagated to the environment.
To make a variable environment variable so that all subprocesses inherit the variable i.e. make the variable available in processes' environment, the usual way on any POSIX shell is to export the variable:
export test=4; python test.py
In your last case:
$ test=2
$ test=4 echo $test
2
the expansion of variable test happening before the echo built-in is run.
You need to use some method to preserve the expansion for later:
$ test=2
$ test=4 sh -c 'echo $test'
4

You need to export the variable for Python.
$ export test=4
Then execute your Python script:
$ ./test.py

This ...
test=4 python test.py
... is a single python command, with variable test explicitly set in its environment, whereas this ...
test=4; python test.py
... is two separate commands. The first tells bash to set variable test (without marking it for export) in the current shell, and the second is the python command. Naturally, Python will not see the variable in its environment. But if you afterward do
echo $test
then the the shell (not the echo command) expands the variable reference to its value as it processes the command line. The resulting expanded command is
echo 4
, which does what you would expect.

Get Python 3 from bash file using similar syntax for Python 3

I am writing an install script and is trying to get a variable linking to Python 3 instead of Python 2. In bash you can do:
:~$ export PYTHON="${PYTHON:-python}"
:~$ echo $PYTHON
/usr/bin/python2
How would you call similar syntax for Python 3 instead? I have tried
:~$ export PYTHON="${PYTHON:-python3}"
but it doesn't work.

Your original, Python 2 code doesn't do what you think it's doing:
:~$ export PYTHON="${PYTHON:-python}"
:~$ echo $PYTHON
/usr/bin/python2
This is because ${foo:-bar} expands the variable named foo, or -- if that variable is either unset or set to an empty value -- expands instead to the value bar as a default. If the result is not bar, then that means that $foo must already be set to a non-bar value.
The logic of this command is roughly thus:
If no shell variable named PYTHON exists, or such a variable exists but with an empty value, then export an environment variable -- and also set a shell variable -- named PYTHON, with a value being python. (Not /usr/bin/python2, but exactly python, those six precise characters).
If a shell variable named PYTHON does exist with a nonempty value, then export it with its current value to the environment. (If that variable is already exported, then this line has no effect).
Since echo $PYTHON is emitting as output /usr/bin/python2, then this means that PYTHON=/usr/bin/python2 must have been true before your original command was invoked at all.
If, unlike your original command, you want to disregard the original value of the PYTHON shell variable (or environment variable, if it's already been exported), then you should do something like the following:
PYTHON=$(type -P python3) || {
echo "python3 not found" >&2
exit 1
}
export PYTHON
The export is done as a separate command here so that the exit status of export does not override the exit status of type, which would prevent errors from being detected.

Why can't environmental variables set in python persist?

I was hoping to write a python script to create some appropriate environmental variables by running the script in whatever directory I'll be executing some simulation code, and I've read that I can't write a script to make these env vars persist in the mac os terminal. So two things:
Is this true?
and
It seems like it would be a useful things to do; why isn't it possible in general?

You can't do it from python, but some clever bash tricks can do something similar. The basic reasoning is this: environment variables exist in a per-process memory space. When a new process is created with fork() it inherits its parent's environment variables. When you set an environment variable in your shell (e.g. bash) like this:
export VAR="foo"
What you're doing is telling bash to set the variable VAR in its process space to "foo". When you run a program, bash uses fork() and then exec() to run the program, so anything you run from bash inherits the bash environment variables.
Now, suppose you want to create a bash command that sets some environment variable DATA with content from a file in your current directory called ".data". First, you need to have a command to get the data out of the file:
cat .data
That prints the data. Now, we want to create a bash command to set that data in an environment variable:
export DATA=`cat .data`
That command takes the contents of .data and puts it in the environment variable DATA. Now, if you put that inside an alias command, you have a bash command that sets your environment variable:
alias set-data="export DATA=`cat .data`"
You can put that alias command inside the .bashrc or .bash_profile files in your home directory to have that command available in any new bash shell you start.

One workaround is to output export commands, and have the parent shell evaluate this..
thescript.py:
import pipes
import random
r = random.randint(1,100)
print("export BLAHBLAH=%s" % (pipes.quote(str(r))))
..and the bash alias (the same can be done in most shells.. even tcsh!):
alias setblahblahenv="eval $(python thescript.py)"
Usage:
$ echo $BLAHBLAH
$ setblahblahenv
$ echo $BLAHBLAH
72
You can output any arbitrary shell code, including multiple commands like:
export BLAHBLAH=23 SECONDENVVAR='something else' && echo 'everything worked'
Just remember to be careful about escaping any dynamically created output (the pipes.quote module is good for this)

If you set environment variables within a python script (or any other script or program), it won't affect the parent shell.
Edit clarification:
So the answer to your question is yes, it is true.
You can however export from within a shell script and source it by using the dot invocation
in fooexport.sh
export FOO="bar"
at the command prompt
$ . ./fooexport.sh
$ echo $FOO
bar

It's not generally possible. The new process created for python cannot affect its parent process' environment. Neither can the parent affect the child, but the parent gets to setup the child's environment as part of new process creation.
Perhaps you can set them in .bashrc, .profile or the equivalent "runs on login" or "runs on every new terminal session" script in MacOS.
You can also have python start the simulation program with the desired environment. (use the env parameter to subprocess.Popen (http://docs.python.org/library/subprocess.html) )
import subprocess, os
os.chdir('/home/you/desired/directory')
subprocess.Popen(['desired_program_cmd', 'args', ...], env=dict(SOMEVAR='a_value') )
Or you could have python write out a shell script like this to a file with a .sh extension:
export SOMEVAR=a_value
cd /home/you/desired/directory
./desired_program_cmd
and then chmod +x it and run it from anywhere.

What I like to do is use /usr/bin/env in a shell script to "wrap" my command line when I find myself in similar situations:
#!/bin/bash
/usr/bin/env NAME1="VALUE1" NAME2="VALUE2" ${*}
So let's call this script "myappenv". I put it in my $HOME/bin directory which I have in my $PATH.
Now I can invoke any command using that environment by simply prepending "myappenv" as such:
myappenv dosometask -xyz
Other posted solutions work too, but this is my personal preference. One advantage is that the environment is transient, so if I'm working in the shell only the command I invoke is affected by the altered environment.
Modified version based on new comments
#!/bin/bash
/usr/bin/env G4WORKDIR=$PWD ${*}
You could wrap this all up in an alias too. I prefer the wrapper script approach since I tend to have other environment prep in there too, which makes it easier for me to maintain.

As answered by Benson, but the best hack-around is to create a simple bash function to preserve arguments:
upsert-env-var (){ eval $(python upsert_env_var.py $*); }
Your can do whatever you want in your python script with the arguments. To simply add a variable use something like:
var = sys.argv[1]
val = sys.argv[2]
if os.environ.get(var, None):
print "export %s=%s:%s" % (var, val, os.environ[var])
else:
print "export %s=%s" % (var, val)
Usage:
upsert-env-var VAR VAL

As others have pointed out, the reason this doesn't work is that environment variables live in a per-process memory spaces and thus die when the Python process exits.
They point out that a solution to this is to define an alias in .bashrc to do what you want such as this:
alias export_my_program="export MY_VAR=`my_program`"
However, there's another (a tad hacky) method which does not require you to modify .bachrc, nor requires you to have my_program in $PATH (or specify the full path to it in the alias). The idea is to run the program in Python if it is invoked normally (./my_program), but in Bash if it is sourced (source my_program). (Using source on a script does not spawn a new process and thus does not kill environment variables created within.) You can do that as follows:
my_program.py:
#!/usr/bin/env python3
_UNUSED_VAR=0
_UNUSED_VAR=0 \
<< _UNUSED_VAR
#=======================
# Bash code starts here
#=======================
'''
_UNUSED_VAR
export MY_VAR=`$(dirname $0)/my_program.py`
echo $MY_VAR
return
'''
#=========================
# Python code starts here
#=========================
print('Hello environment!')
Running this in Python (./my_program.py), the first 3 lines will not do anything useful and the triple-quotes will comment out the Bash code, allowing Python to run normally without any syntax errors from Bash.
Sourcing this in bash (source my_program.py), the heredoc (<< _UNUSED_VAR) is a hack used to "comment out" the first-triple quote, which would otherwise be a syntax error. The script returns before reaching the second triple-quote, avoiding another syntax error. The export assigns the result of running my_program.py in Python from the correct directory (given by $(dirname $0)) to the environment variable MY_VAR. echo $MY_VAR prints the result on the command-line.
Example usage:
$ source my_program.py
Hello environment!
$ echo $MY_VAR
Hello environment!
However, the script will still do everything it did before except exporting, the environment variable if run normally:
$ ./my_program.py
Hello environment!
$ echo $MY_VAR
<-- Empty line

As noted by other authors, the memory is thrown away when the Python process exits. But during the python process, you can edit the running environment. For example:
>>> os.environ["foo"] = "bar"
>>> import subprocess
>>> subprocess.call(["printenv", "foo"])
bar
0
>>> os.environ["foo"] = "foo"
>>> subprocess.call(["printenv", "foo"])
foo
0