I have a shell script that executes some python code, and installs the necessary dependencies to do so beforehand:
sudo -E PATH="$PATH" python3 -m pip install -r requirements.txt
The script runs successfully when executed like so:
./script.sh
but I get an import error from Python when I run the script with sudo permission:
sudo ./script.sh
Traceback (most recent call last):
File "dsc.py", line 2, in <module>
from imblearn.over_sampling import ADASYN
File "/usr/local/lib/python3.8/dist-packages/imblearn/__init__.py", line 53, in <module>
from . import ensemble
File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/__init__.py", line 8, in <module>
from ._forest import BalancedRandomForestClassifier
File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/_forest.py", line 28, in <module>
from sklearn.utils.fixes import _joblib_parallel_args
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/usr/local/lib/python3.8/dist-packages/sklearn/utils/fixes.py)
I don't understand why Python can no longer import ADASYN when the script is given sudo permission.
I am using virtualenv for version control. When I check where the imblearn package is located when in the same virtual environment from where the shell script was exectued, I get:
❯ python3 -m pip show imblearn
WARNING: Package(s) not found: imblearn
But I can find it when not in a virtual environment:
❯ deactivate
❯ python3 -m pip show imblearn
Name: imblearn
Version: 0.0
Summary: Toolbox for imbalanced dataset in machine learning.
Home-page: https://pypi.python.org/pypi/imbalanced-learn/
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /usr/local/lib/python3.8/dist-packages
Requires: imbalanced-learn
Required-by:
Which means that my shell script does install the dependencies, but not within the virtual environment from which it is executed. Removing the imblearn package and running the shell script installs it again, so I am certain that it is this shell script that installs the package to the default location. I thought that by running Python code with the sudo -E PATH="$PATH" option makes sure that the python command has the same path as the virtual environment from which it is run, and thus the correct dependencies. But given that the dependencies get installed in my default Python package directory /usr/local/lib/python3.8/dist-packages, this seems not to be the case.
So how can I run python code in a shell script within the same virtual environment from which the shell script was executed? Is this possible at all? I am familiar with Python, but very new to shell.
I'm using Python 3.8.10 (both as default and in the virtualenv), Ubuntu 20.04 and zsh.
If you run sudo ./script.sh, the actually executor of the script is the superuser, which means none of the env-related stuff in your virtual env is synced over.
You may need to specify the python virtual env inside your ./script.sh, i.e., adding one line on the top of the ./script.sh: #!path/to/your/virtualenvs/python
The problem can be mitigated by running the bash script like so:
sudo -E PATH=$PATH ./script.sh
instead of just sudo ./script.sh.
This way, sudo executes the shell script, and the python path of your current virtualenv is passed along.
Inside the shell script, you now don't need to execute all python scripts with sudo -E PATH=$PATH script.py, but simply using python script.py suffices, since in this case python refers to the virtualenv python version.
Related
I'm trying to create new Python virtual environment, but I have problems with pip.
I want my virtual environment to run on Python 3.10, but my default system Python version is 3.9.5. So I downloaded Python 3.10 with
sudo apt-get install python3.10
and then i run
python3 -m virtualenv venv --python=python3.10
to create virtual environment.
The problem is, when I try to use pip I get an error:
Traceback (most recent call last):
File "/home/<my user name>/venv/bin/pip", line 5, in <module>
from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
I tried running
python -m ensurepip --upgrade
but got an error:
/home/pablo/venv/bin/python: No module named ensurepip
Also running whereis pip shows, that there are some pips in the virtual environment.
pip: /usr/bin/pip /home/<my user name>/venv/bin/pip3.10 /home/<my user name>/venv/bin/pip /usr/share/man/man1/pip.1.gz
Also, when I create virtual environments with my default python interpreter, everything works perfectly fine.
I would recommend using pyenv with pyenv-virtualenv. In my experience it's just too much work to go manual. Sorry I know this is not a true answer, but having wasted hours and hours on similar issues, I feel like sharing )
your installed binary executable is named as pip3.10(/usr/bin/pip /home//venv/bin/pip3.10). So you should use the same name to call it, just pip wouldn't work. Have you tried that?
I am currently using Fabric3 in a python project and I want to use the azure-cli package. However when I install the package, it appears to break my Fabric3 install:
root ~ $ fab --version
Traceback (most recent call last):
File "/opt/rh/rh-python36/root/usr/bin/fab", line 7, in
from Fabric3.main import program
ModuleNotFoundError: No module named 'Fabric3'
I noticed that the pip install of azure-cli appears to install fabric (2.5.0), and this appears to overwrite some of the fabric files within the "/opt/rh/rh-python36/root/usrbin/fab" script and the "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/fabric/" directory.
So my question is, is there a way to run different versions of Fabric within the same project?
Thanks in advance
I found a way to do this by using python virtual environments.
I created a virtual env, activated the virtual env, installed the pip package, deactivated the virtual env, then used Fabric 3 to call the function in the venv.
python -m venv azure_cli_venv
. azure_cli_venv/bin/activate
pip install azure-cli
deactivate
Fabric3 local call:
local(f'./azure_cli_venv/bin/az --help
I have both Python 3.7 and 3.6 installed. But in my profile I set to use 3.6 and this is indeed a version that gets reported by which python3.
$ which python3
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
I then create virtual env by running python3 -m venv venv
And everything in my venv gets linked to the 3.6. I confirm this with ls -la venv/bin and by inspecting venv/pyvenv.cfg.
I activate the venv virtual environment.
I then install a zappa library with pip install zappa.
And when I run zappa deploy, I get an error that Python 3.7 is not supported. Why would a library use the version of Python that is not configured within the virtualenv and how do I make it use the correct version?
(venv) $ zappa deploy
Traceback (most recent call last):
File "/Users/t/Projects/djzappa/Spheres/venv/bin/zappa", line 7, in <module>
from zappa.cli import handle
File "/Users/t/Projects/djzappa/Spheres/venv/lib/python3.7/site-packages/zappa/__init__.py", line 12, in <module>
raise RuntimeError(err_msg)
RuntimeError: This version of Python (3.7) is not supported!
Zappa (and AWS Lambda) support the following versions of Python: ['2.7', '3.6']
NOTE: I do NOT use virtualenv as everybody is suggesting, and as other existing answers are referring.
Use this when you create environment:
virtualenv --python=/usr/bin/python3.6
But if you want to use venv, first of all, check whether python3 in your terminal refers to the python3.6 as you want.
If not - run your command with the full path to python3.6 instead of sole python3 in python3 -m venv venv.
Edit: why your alias python3 is referring to python3.7 - it probably depends on OS you use and sequence of installation. Hard to say.
As an addition to the accepted answer, be also aware that changing the directory name where your venv is located causes using the default python and pip paths of your system, instead of using the venv one.
The Python version is different from the virtualenv python version and that is why it is complaining.
You can recreate your virtual environment(delete the existing one) and use the following command
virtualenv --python=your python version here venv
Or
mkvirtualenv venv --python=your python version here
if you want to create your virtual environment using mkvirtualenv command.
What is the difference between:
python -m pip install forecasting
and
pip install forecasting
In my environment, the first is working when the second format raise the following error:
ModuleNotFoundError: No module named 'pip._internal'
Answer to your first question.
From pip docs
pip is a command line program. When you install pip, a pip command is added to your system, which can be run from the command prompt as follows:
$ pip <pip arguments>
If you cannot run the pip command directly (possibly because the location where it was installed isn't on your operating system's PATH) then you can run pip via the Python interpreter:
$ python -m pip <pip arguments>
With reference to this GitHub issue:
https://github.com/pypa/pip/issues/5373
Try the following command:
sudo easy_install pip
They're two different ways a package can expose commands to the commandline.
pip is a console_script entry-point. Any package can define globally available commands that way, and PIP (the package) uses it to define pip (the command).
In case of pip, the function they're executing using this method is set to pip._internal.main():
entry_points={
"console_scripts": [
"pip=pip._internal:main",
],
},
On the other side python -m pip is using a switch for calling modules. If your module contains a __main__.py file, this file will simply be interpreted and executed by the Python interpreter.
In case of python -m pip, this file essentially contains
from pip._internal import main as _main
if __name__ == '__main__':
sys.exit(_main())
so the two commands try to do the same.
However, recently PIP has shown some weird quirks [1] [2] that cause one of the two to work, and the other one to fail.
I want to run a python script in in-built anaconda environment tensorflow_p36. To check if it is in virtual environment or not, I am using command pip -V.
My first attempt at bash script:
#!/bin/bash
source activate tensorflow_p36
python /home/ec2-user/abc/temp.py
pip -V
Note: tensorflow_p36 being an in-built environment, does not require to be called from specific /env/bin directory. It can be activated from any directory. I think it's a feature of Amazon Deep Learning AMIs.
My second attempt at bash script:
#!/bin/bash
pythonEnv="/home/ec2-user/anaconda3/envs/tensorflow_p36/"
source ${pythonEnv}bin/activate
${pythonEnv}bin/python /home/ec2-user/abc/temp.py
pip -V
Note: When I try to run source /home/ec2-user/anaconda3/envs/tensorflow_p36/bin/activate command in terminal, the environment isn't being activated.
Each time, I am getting the same result:
pip 9.0.1 from /home/ec2-user/anaconda3/lib/python3.6/site-packages (python 3.6)
Whereas, I should be getting:
pip 9.0.1 from /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (python 3.6)
Can someone please explain how do I activate virtual environment and run a python script from that environment? I need to use this particular environment because of the dependencies installed in it.
Extra info:
Not sure if it matters, but the tensorflow_p36 is a conda environment, not a virtualenv.
This works with virtualenv. Create environment:
virtualenv -p python 3.6 tensorflow_p36
Then change the script to:
#!/bin/bash
source $HOME/tensorflow_p36/bin/activate
python /home/ec2-user/abc/temp.py
I believe the confusion has to do with the fact that you are using anaconda and not virtualenv to create a python environment. These two tools work differently.
If you are using an EC2 instance, why not to install tensorflow_p36 globally anyway?