Develop Reference

modifying the PYTHONPATH variable in zsh

I would like to add my site-packages directory to the PYTHONPATH variable, so I can import modules which are stored there to use them in scripts in interpreter. My default shell is zsh, and the OS is macOS Monterey, version 12.6.
I have tried to open the ~/.zprofile in nano editor from my terminal, adding the path I need, and restarting my terminal, but it did not help to solve the issue. After restarting the terminal, I tried to run following commands:
narynaa#Narynas-MacBook-Pro ~ % echo ~/.z.profile
/Users/narynaa/.z.profile
narynaa#Narynas-MacBook-Pro ~ % cat ~/.zprofile
# Set PATH, MANPATH, etc., for Homebrew.
eval "$(/opt/homebrew/bin/brew shellenv)"
export PYTHONPATH="/opt/homebrew/lib/python3.10/site-packages"
The path I need is stored inside the ~/.zprofile file now, but the problem is not solved.
Thank you for your help!

How to get python via virtual environment recognized in bash file via shebang

I have a script file:
#!/usr/bin/env python3.9
print("python is working")
However when I try and run it:
(karl-env) karl#Karls-MBP scripts (karl/test) $ . test.sh
bash: test.sh: line 3: syntax error near unexpected token `"python is working"'
bash: test.sh: line 3: `print("python is working")'
Following info:
(karl-env) karl#Karls-MBP scripts (karl/test) $ type -a python
python is /Users/karl/.pyenv/shims/python
python is /Users/karl/.pyenv/shims/python
python is /usr/bin/python
I'm in a virtual environment but I fail to understand how to get my environments python recognized via the shebang #!/usr/bin/env python3.9. I do not use Python often hence my noobiness!

This has a little to do with Python and a lot to do with the shell.
You're doing . test.sh – . is an alias for source, which has your shell attempt to interpret the given script as shell commands you'd enter. You want ./test.sh to execute the script.
Your shebang line is explicitly looking for a python3.9 executable, and your environment might not be Python 3.9, so you fall back to something else. Do python (or python3) instead: #!/usr/bin/env python
For the sake of sanity, rename your script to .py; it's not a .shellscript.

What shebang to use for Python scripts run under a pyenv virtualenv

When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.

I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.

As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)

If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.

To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.

It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())

If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python

/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py

Maybe you need to check the file privileges:
sudo chmod +x script.py

Set Matplotlib backend when using Windows Subsystem for Linux

My Matplotlib backend keeps reverting to TkAgg. This is a problem because in the Windows Subsystem for Linux (WSL), you can't do GUI stuff, and so I get the error
TclError: no display name and no $DISPLAY environment variable
I've tried adding a matplotlibrc file to /home/<user>/.config/matplotlib (in the Windows filesystem, this is C:\Users\<user>\AppData\Local\lxss\home\<user>\.config\matplotlib).
My matplotlibrc looks like this
backend : Agg
However, if I do this
$ cd /home/<user>/.config/matplotlib
$ ls -A
nothing shows up.
When I try
$ python
>>> import matplotlib
>>> matplotlib.get_backend()
'TkAgg'
so clearly it's not setting the backend to Agg. Why not?
Update:
I've figured out that putting backend : Agg in C:\Users\<user>\AppData\Local\lxss\home\<user>\.config\matplotlib\matplotlibrc changes the backend in Windows Python only, leaving Linux Python as TkAgg. Which is odd, because Windows Python should only be using C:\Users\<user>\AppData\Local\Enthought\Canopy\User\Lib\site-packages\matplotlib\mpl-data\matplotlibrc, right?

I figured it out. I had created the matplotlibrc file using Windows - this is why it didn't show up when I tried $ ls -A in bash. So I did this instead (after deleting the Windows-created file):
$ cd /home/<user>/.config/matplotlib
$ touch matplotlibrc
$ echo "backend : Agg" > matplotlibrc
(touch matplotlibrc creates an empty matplotlibrc file)
This did the trick, and my Windows python matplotlib's backend was left as Qt4Agg.
Here it is on one line for copy/paste:
cd /home/<user>/.config/matplotlib; touch matplotlibrc && echo "backend : Agg" > matplotlibrc

using python virtual env in R

I am using 'rPython' package for calling python within R but I am unable to make R refer to my python's virtual environment.
In R, I have tried using
system('. /home/username/Documents/myenv/env/bin/activate')
but after running the above my python library path does not change (which I check via python.exec(print sys.path)). When I run
python.exec('import nltk')
I am thrown the error:
Error in python.exec("import nltk") : No module named nltk
although it is there in my virtual env.
I am using R 3.0.2, Python 2.7.4 on Ubuntu 13.04.
Also, I know I can change the python library path from within R by using
python.exec("sys.path='\your\path'")
but I don't want this to be entered manually over and over again whenever a new python package is installed.
Thanks in advance!

Use the "activate" bash script before running R, so that the R process inherits the changed environment variables.
$ source myvirtualenv/bin/activate
$ R
Now rPython should be able to use the packages in your virtualenv.
Works for me. May behave strangely if the Python version you made the virtualenv with is different to the one rPython links into the R process.

Expanding on #PaulHarrison's answer, you can mimic what .../activate is doing directly in the environment (before starting python from R).
Here's one method for determining what vars are modified:
$ set > pyenv-pre
$ . /path/to/venv/activate
(venvname) $ set > pyenv-post
(venvname) $ diff -uw pyenv-pre pyenv-post
This gave me something like:
--- pyenv-pre 2018-12-02 15:16:43.093203865 -0800
+++ pyenv-post 2018-12-02 15:17:34.084999718 -0800
## -33,10 +33,10 ##
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+PATH=/path/to/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PIPESTATUS=([0]="0")
PPID=325990
-PS1='\[\e]0;\u#\h: \w\a\]${debian_chroot:+($debian_chroot)}\u#\h:\w\$ '
+PS1='(venvname) \[\e]0;\u#\h: \w\a\]${debian_chroot:+($debian_chroot)}\u#\h:\w\$ '
PS2='> '
PS4='+ '
PWD=/
## -50,10 +50,13 ##
TERM=xterm
UID=3000019
USER='helloworld'
+VIRTUAL_ENV=/path/to/venv
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/3000019
XDG_SESSION_ID=27577
-_=set
+_=/path/to/venv/bin/activate
+_OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+_OLD_VIRTUAL_PS1='\[\e]0;\u#\h: \w\a\]${debian_chroot:+($debian_chroot)}\u#\h:\w\$ '
__git_printf_supports_v=yes
__grub_script_check_program=grub-script-check
_backup_glob='#(#*#|*#(~|.#(bak|orig|rej|swp|dpkg*|rpm#(orig|new|save))))'
## -2390,6 +2393,31 ##
fi;
fi
}
+deactivate () ... rest of this function snipped for brevity
So it appears that the important envvars to update are:
PATH: prepend the venv bin directory to the existing paths
VIRTUAL_ENV: set to /path/to/venv
I believe the other changes (OLD_VIRTUAL_* and deactivate () ...) are optional and really only used to back-out the venv activation.
Looking at the .../activate script verifies these are most of the steps taken. Another step is unset PYTHONHOME if set, which may not be shown in the diff above if you didn't have it set previously.
To R-ize this:
Sys.setenv(
PATH = paste("/path/to/venv/bin", Sys.getenv("PATH"), sep = .Platform$path.sep),
VIRTUAL_ENV = "/path/to/venv"
)
Sys.unsetenv("PYTHONHOME") # works whether previously set or not

I've had luck getting scripts to use my peynv installation by using:
#!/usr/bin/env python
So maybe try pointing R to that path (sans #!, of course).

manage to get it working by using bash -c:
system("/bin/bash -c \"source ./pydatatable/py-pydatatable/bin/activate && python -c 'import datatable as dt; print(dt.__version__)'\"")