Develop Reference

How to run shell-exectued python script in a virtual environment

I have a shell script that executes some python code, and installs the necessary dependencies to do so beforehand:
sudo -E PATH="$PATH" python3 -m pip install -r requirements.txt
The script runs successfully when executed like so:
./script.sh
but I get an import error from Python when I run the script with sudo permission:
sudo ./script.sh
Traceback (most recent call last):
File "dsc.py", line 2, in <module>
from imblearn.over_sampling import ADASYN
File "/usr/local/lib/python3.8/dist-packages/imblearn/__init__.py", line 53, in <module>
from . import ensemble
File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/__init__.py", line 8, in <module>
from ._forest import BalancedRandomForestClassifier
File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/_forest.py", line 28, in <module>
from sklearn.utils.fixes import _joblib_parallel_args
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/usr/local/lib/python3.8/dist-packages/sklearn/utils/fixes.py)
I don't understand why Python can no longer import ADASYN when the script is given sudo permission.
I am using virtualenv for version control. When I check where the imblearn package is located when in the same virtual environment from where the shell script was exectued, I get:
❯ python3 -m pip show imblearn
WARNING: Package(s) not found: imblearn
But I can find it when not in a virtual environment:
❯ deactivate
❯ python3 -m pip show imblearn
Name: imblearn
Version: 0.0
Summary: Toolbox for imbalanced dataset in machine learning.
Home-page: https://pypi.python.org/pypi/imbalanced-learn/
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /usr/local/lib/python3.8/dist-packages
Requires: imbalanced-learn
Required-by:
Which means that my shell script does install the dependencies, but not within the virtual environment from which it is executed. Removing the imblearn package and running the shell script installs it again, so I am certain that it is this shell script that installs the package to the default location. I thought that by running Python code with the sudo -E PATH="$PATH" option makes sure that the python command has the same path as the virtual environment from which it is run, and thus the correct dependencies. But given that the dependencies get installed in my default Python package directory /usr/local/lib/python3.8/dist-packages, this seems not to be the case.
So how can I run python code in a shell script within the same virtual environment from which the shell script was executed? Is this possible at all? I am familiar with Python, but very new to shell.
I'm using Python 3.8.10 (both as default and in the virtualenv), Ubuntu 20.04 and zsh.

If you run sudo ./script.sh, the actually executor of the script is the superuser, which means none of the env-related stuff in your virtual env is synced over.
You may need to specify the python virtual env inside your ./script.sh, i.e., adding one line on the top of the ./script.sh: #!path/to/your/virtualenvs/python

The problem can be mitigated by running the bash script like so:
sudo -E PATH=$PATH ./script.sh
instead of just sudo ./script.sh.
This way, sudo executes the shell script, and the python path of your current virtualenv is passed along.
Inside the shell script, you now don't need to execute all python scripts with sudo -E PATH=$PATH script.py, but simply using python script.py suffices, since in this case python refers to the virtualenv python version.

venv and pipenv not producing the same environment

While trying to build the documents of pydantic I came across a problem setting up a virtual environment using pipenv.
Following the steps described in here by samuelcolvin worked just fine:
# Using pyenv + vevn
cd code/pydantic
git checkout master
git pull
pyenv install 3.9.13
pyenv shell 3.9.13
python3 -m venv env-3.9.13
source env-3.9.13/bin/activate
make install
make install-docs
make docs
However, trying to set up the environment using pipenv fails. The steps I took are:
# Using pyenv + pipenv
pyenv install 3.9.13
pipenv --python 3.9
pipenv shell
make install
make install-docs
make docs
The last part fails with the following error:
flake8 --max-line-length=80 docs/examples/
python docs/build/main.py
error in hypothesis_property_based_test.py: Traceback (most recent call last):
File "/Users/drorata/dev/pydantic/docs/examples/hypothesis_property_based_test.py", line 2, in <module>
from hypothesis import given, strategies as st
AttributeError: type object 'Path' has no attribute '_flavour'
1 errors, not writing files
make: *** [docs] Error 1
I appreciate that this is a very specific issue, but maybe by solving it me and others could learn something important about pipenv.

Pip is not found after creating new python virtual environment

I'm trying to create new Python virtual environment, but I have problems with pip.
I want my virtual environment to run on Python 3.10, but my default system Python version is 3.9.5. So I downloaded Python 3.10 with
sudo apt-get install python3.10
and then i run
python3 -m virtualenv venv --python=python3.10
to create virtual environment.
The problem is, when I try to use pip I get an error:
Traceback (most recent call last):
File "/home/<my user name>/venv/bin/pip", line 5, in <module>
from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
I tried running
python -m ensurepip --upgrade
but got an error:
/home/pablo/venv/bin/python: No module named ensurepip
Also running whereis pip shows, that there are some pips in the virtual environment.
pip: /usr/bin/pip /home/<my user name>/venv/bin/pip3.10 /home/<my user name>/venv/bin/pip /usr/share/man/man1/pip.1.gz
Also, when I create virtual environments with my default python interpreter, everything works perfectly fine.

I would recommend using pyenv with pyenv-virtualenv. In my experience it's just too much work to go manual. Sorry I know this is not a true answer, but having wasted hours and hours on similar issues, I feel like sharing )

your installed binary executable is named as pip3.10(/usr/bin/pip /home//venv/bin/pip3.10). So you should use the same name to call it, just pip wouldn't work. Have you tried that?

pipenv, jupyter and possibly other commands stopped working after upgrading python3.7 to python3.8

I upgraded python version from 3.7 to 3.8 using brew upgrade python3 however after upgrading pipenv and jupyter commands stopped working. These are the commands I've ran as of now other commands may not be working as well.
This is the output when I ran pipenv:
-bash: /usr/local/bin/pipenv: /usr/local/opt/python/bin/python3.7: bad interpreter: No such file or directory
This is the output when I ran the jupyter command:
-bash: /usr/local/bin/jupyter: /usr/local/opt/python/bin/python3.7: bad interpreter: No such file or directory
So I opened up the file /usr/local/bin/pipenv and the interpreter declared at the top is #!/usr/local/opt/python/bin/python3.7
So I changed it to #!/usr/local/opt/python/bin/python3.8 and ran pipenv command and this is the output:
Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 5, in <module>
from pipenv import cli
ModuleNotFoundError: No module named 'pipenv'
I ran pip3 list command and this was the output
Package Version
---------- -------
pip 20.1.1
setuptools 49.2.0
wheel 0.34.2
So i went into /usr/local/lib and there were 2 folders named python3.7 and python3.8. The site-packages folder in python3.7 has all the modules I had previously installed but the site-packages folder in python3.8 contains only 3 modules which I got using pip3 list.
I don't fully understand the issue but I think all the older commands are still linked to python3.7 interpreter which no longer exists and secondly all the site packages need to be in /usr/local/lib/python3.8

Yes, site-packages are per-interpreter-version. (Also, virtualenvs tend to break when you upgrade your Python version.)
Just reinstall the packages you need with your new Python 3.8 based pip3.

Python venv (virual environment) uses wrong version of Python

I have both Python 3.7 and 3.6 installed. But in my profile I set to use 3.6 and this is indeed a version that gets reported by which python3.
$ which python3
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
I then create virtual env by running python3 -m venv venv
And everything in my venv gets linked to the 3.6. I confirm this with ls -la venv/bin and by inspecting venv/pyvenv.cfg.
I activate the venv virtual environment.
I then install a zappa library with pip install zappa.
And when I run zappa deploy, I get an error that Python 3.7 is not supported. Why would a library use the version of Python that is not configured within the virtualenv and how do I make it use the correct version?
(venv) $ zappa deploy
Traceback (most recent call last):
File "/Users/t/Projects/djzappa/Spheres/venv/bin/zappa", line 7, in <module>
from zappa.cli import handle
File "/Users/t/Projects/djzappa/Spheres/venv/lib/python3.7/site-packages/zappa/__init__.py", line 12, in <module>
raise RuntimeError(err_msg)
RuntimeError: This version of Python (3.7) is not supported!
Zappa (and AWS Lambda) support the following versions of Python: ['2.7', '3.6']
NOTE: I do NOT use virtualenv as everybody is suggesting, and as other existing answers are referring.

Use this when you create environment:
virtualenv --python=/usr/bin/python3.6
But if you want to use venv, first of all, check whether python3 in your terminal refers to the python3.6 as you want.
If not - run your command with the full path to python3.6 instead of sole python3 in python3 -m venv venv.
Edit: why your alias python3 is referring to python3.7 - it probably depends on OS you use and sequence of installation. Hard to say.

As an addition to the accepted answer, be also aware that changing the directory name where your venv is located causes using the default python and pip paths of your system, instead of using the venv one.

The Python version is different from the virtualenv python version and that is why it is complaining.
You can recreate your virtual environment(delete the existing one) and use the following command
virtualenv --python=your python version here venv
Or
mkvirtualenv venv --python=your python version here
if you want to create your virtual environment using mkvirtualenv command.