I'm a strong pyenv and poetry user that's beginning to use pipx and looking to learn more about how it works. Specifically, I'd like to understand how it determines which Python version to use when installing.
I've noticed that it seems to search PATH for existing references to applications when you pipx install <package> to make existing installations accessible globally. This, for instance, works well with pyenv where it pipx will find a version of the package you install across any Python versions installed via pyenv.
pipx install cookiecutter
⚠️ Note: cookiecutter was already on your PATH at ~/.pyenv/shims/cookiecutter
installed package cookiecutter 1.7.3, Python 3.9.6
These apps are now globally available
- cookiecutter
done! ✨ 🌟 ✨
But what if it's a package you've never installed before? And there's a package version compatible with 3.6, 3.7, 3.8 – how will it determine with Python version to use when installing this package?
I've found you can provide the Python version PipX should install the package into by providing it as an argument to the install command (see below).
More specifically, when using pyenv, you can switch to the version you desire to install into and then provide $(which python) to automatically provide the path to that python version...
pyenv shell 3.X.X
pipx install <package-name> --python $(which python)
Related
Is setuptools always installed with Python?
I would like to invoke setuptools at runtime outside of a setup.py script.
In other words, should I include setuptools inside my package's requirements.txt and setup.py's install_requires list?
Background
I have noticed when creating a new virtual environment (with Python 3.7.9) that both pip and setuptools are installed by default:
python -m venv venv
source ./venv/bin/activate
pip list
Package Version
---------- -------
pip 20.1.1
setuptools 47.1.0
This is documented here: Creating Virtual Environments:
venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.
Even in a vanilla version of Python 3.7.6 (installed via pyenv), the packages installed by default are both pip and setuptools.
Research
Should setuptools be included in setup_requires in Python?
Informs that setuptools should not be included in setup_requires, but does not talk about it being included in package requirements for runtime use.
TL;DR
Formally, No. Usually, Yes.
the setuptools is not part of the python vanilla codebase, hence not a vanilla modules.
python.org installers or mac homebrew will install it for you, but if someone compile the python by himself or install it on some linux distribution he may not get it and will need to install it by himself.
I am using macOS X and I installed native Python 3.8 in April, and the packages of this version are stored in:
/Users/hubert/Library/Python/3.8/lib/python/site-packages/
Some weeks later, in May, I installed Anaconda, but it uses a different path for it's packages:
/Users/hubert/opt/anaconda3/lib/python3.7/site-packages/
Today I needed to install a new package to be used in a native Python script. I typed:
pip install ffmpeg
I've got a success-message. But when I wanted to run my script, it couldn't find ffmpeg. I rebooted my Mac, but the script still couldn't find it. So I again tried pip install ffmpeg. Now I did get this message:
Requirement already satisfied: ffmpeg in ./opt/anaconda3/lib/python3.7/site-packages (1.4)
But obviously native Pythons wants to have it in the other path.
My questions:
Is it save to just copy the directories ffmpeg and ffmpeg-1.4.dist-info from the anaconda directory to the native directory?
How can I tell pip to use the native Python directory to install new packages?
Is it a good idea to merge the two site-packages directories? And if it's a good idea: What is the best way to let native Python 3.8 and Anaconda (which obviously uses Python 3.7) use the same site-packages directory?
Is it a good idea to modify site-packages manually?
Probably not. The problems you're having are the sorts of problems that virtual environment managers like Anaconda were created to avoid. If you start messing with site-packages, at a minimum you run the risk of creating a lot of new problems in your dependency tree. Unless you need to use two different python installations in the same shell prompt at the same time, your life will be much easier if you just use Anaconda to manage all your environments.
(Bear in mind that creating a new installation of Python 3.8 with ffmpeg installed is as easy as conda create -n mynewenv python=3.8 ffmpeg. Then run conda activate mynewenv whenever you want to use Python 3.8 with ffmpeg installed.)
How to install packages in the right place with pip
Use the right copy of pip
The easiest way is to make sure that you're using the right pip. On MacOS or Linux, you can do this by running which python or which pip before you start installing things. If you want to install to native Python, you'll probably have to run conda deactivate before you start running pip commands.
It's always possible that the pip in your search path is actually a symlink, so it's worth checking this with file -h $(which pip). (In my case, I have a couple of versions of Python installed with homebrew, which likes to symlink things into /usr/local/bin.)
Example
(base) ~ $ which python
/miniconda3/bin/python
(base) ~ $ which pip
/miniconda3/bin/pip
(base) ~ $ conda deactivate
~ $ which python
/usr/local/bin/python
~ $ which pip
/usr/local/bin/pip
~ $ file -h $(which pip)
/usr/local/bin/pip: symbolic link to ../Cellar/python#2/2.7.16_1/bin/pip
Note that the pip for your native installation of Python 3.8 is probably called pip3.
Check where pip is sending stuff
You can check to see where pip/pip3 is installing stuff by checking where a specific package was installed. This can be a way to double-check where things will be installed if it isn't obvious from which pip.
Example
~ $ pip list
Package Version
----------------------------- ---------
...
wheel 0.33.4
~ $ pip show wheel
Name: wheel
Version: 0.33.4
Summary: A built-package format for Python.
Home-page: https://github.com/pypa/wheel
Author: Daniel Holth
Author-email: dholth#fastmail.fm
License: MIT
Location: /usr/local/lib/python2.7/site-packages
Requires:
Required-by:
Note the Location line.
I have pyenv installed on an up to date debian testing distribution, and have python 2.7.1 in it.
I create a virtualenv for a project with this version, but when activated it gives me the python system version (3.7).
Here is what I did:
$ pyenv virtualenv 2.7.16 my_project-2.7
Requirement already satisfied: virtualenv in /home/user/.local/lib/python2.7/site-packages (15.1.0)
You are using pip version 18.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Using base prefix '/usr'
New python executable in /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/bin/python
Installing setuptools, pip, wheel...
done.
Installing pip from https://bootstrap.pypa.io/get-pip.py...
Collecting pip
Using cached https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 19.1.1
Uninstalling pip-19.1.1:
Successfully uninstalled pip-19.1.1
Successfully installed pip-19.1.1
$ pyenv activate my_project-2.7
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
$ python --version
Python 3.7.3
$ pip --version
pip 19.1.1 from /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/lib/python3.7/site-packages/pip (python 3.7)
It is strange. The virtualenv is correctly set into ~/.pyenv/version/2.7.16, but I can see that pip come from /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/lib/python3.7/
I think I missed something, or I might have some mess in my python installation, but I can't find out what this python3.7 lib is doing here.
How can I have this virtualenv created for python2.7 ?
Edit
I tried to completely remove pyenv (removing the $(pyenv root) directory), then reinstall it with pyenv-installer but it did not change anything.
I created two virtualenvs for two projects I have (the above 2.7 and one in 3.5).
$ pyenv virtualenvs
2.7.16/envs/my_project-2.7 (created from /usr)
3.6.8/envs/other_project-3.6 (created from /home/user/.pyenv/versions/3.6.8)
my_project-2.7 (created from /usr)
other_project-3.6 (created from /home/user/.pyenv/versions/3.6.8)
I think the problem could be found from this created from /usr but I don't understand what it means. Why was this virtualenv not created from the pyenv python version ? Is it at all relevant ?
It seems that when I create a 3.+ virtual env, venv is used. But on 2.7, virtualenv is used and fails.
WFM as written if I install pyenv and pyenv-virtualenv from scratch according to the linked instructions.
I can see that pip come from /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/lib/python3.7/
This shows that most probably, that virtual environment already exists and has a wrong Python version.
I can only conjecture how it ended up like this. Most probably, you were messing with that virtualenv or with pyenv envvars by hand; or ran scripts/have something in your bash stratup scripts that do that. E.g. if you are using pyenv-virtualenv commands to manage your environments, you are not supposed to mix them with regular virtualenv commands.
Deleting this virtualenv with pyenv virtualenv-delete my_project-2.7 then recreating it should help.
If not, delete ~/.pyenv and everything relevant from your startup scripts, restart the shell and reinstall pyenv and pyenv-virtualenv.
For a less destructive fixing, you need to carefully inspect your shell environment (envvars, aliases, defined shell functions) and startup scripts for anything that might conflict with pyenv's work; if that doesn't help, debug pyenv machinery (export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }' helps here very much) to find out what exactly is happening.
I used pyenv, pyenv-virtualenv for managing python virtual environment.
I have a project working in Python 3.4 virtual environment.
So all installed packages(pandas, numpy etc) are not newest version.
What I want to do is to upgrade Python version from 3.4 to 3.6 as well as upgrade other package version to higher one.
How can I do this easily?
Here is how you can switch to 3.9.0 for a given virtual environement venv-name:
pip freeze > requirements-lock.txt
pyenv virtualenv-delete venv-name
pyenv virtualenv 3.9.0 venv-name
pip install -r requirements-lock.txt
Once everything works correctly you can safely remove the temporary requirements lock file:
rm requirements-lock.txt
Note that using pip freeze > requirements.txt is usually not a good idea as this file is often used to handle your package requirements (not necessarily pip freeze output). It's better to use a different (temporary) file just to be sure.
Use pip freeze > requirements.txt to save a list of installed packages.
Create a new venv with python 3.6.
Install saved packages with pip install -r requirements.txt. When pip founds an universal wheel in its cache it installs the package from the cache. Other packages will be downloaded, cached, built and installed.
OP asked to upgrade the packages alongside Python. No other answers address the upgrade of packages. Lock files are not the answer here.
Save your packages to a requirements file without the version.
pip freeze | cut -d"=" -f1 > requirements-to-upgrade.txt
Delete your environment, create a new one with the upgraded Python version, then install the requirements file.
pyenv virtualenv-delete venv-name
pyenv virtualenv 3.6.8 venv-name
pip install -r requirements-to-upgrade.txt
The dependency resolver in pip should try to find the latest package. This assumes you have the upgrade Python version installed (e.g., pyenv install 3.6.8).
If you use anaconda, just type
conda install python==$pythonversion$
So I heard about the proper way to install packages into python by creating a new virtual environment for every project. Being on a mac (10.8) I have installed python3 using Homebrew, then I installed pip and virtualenv on this copy.
Now here comes the problem:
I create a new virtualenv, and activate it using:
virtualenv testing
source testing/bin/activate
When I type
which python
/Users/mik/Desktop/testing/bin/python
But typing
which pip
/usr/local/bin/pip
(learned of this when trying to install a package in the virtual environment, and it installed in the system wide installation in /usr/local/)
Inside the folder testing there is no file referring to pip
Extra Question: How does pip know which python to install the files to, for example pip list (which I believe refers to python 2.7) outputs the names of packages installed on python 3.3
I'll start with the last question as it explains what is happening.
The commands pip and easy_install are python scripts which are made executable on the filesystem. The python they use is the python that the first line tells to run the script. e.g. in /usr/bin/easy_install it is #!/usr/bin/python This will be Apple's python. So easy_install will install the 2.7 version of pip and virtualenv and will ignore your python3.3 setup.
The way to instal into python 3 is to install the 3.3 version of pip and virtualenv, the easiest way would be to install the Homebrew package for them. I think it is easier and less confusing to use just one package manager (Homebrew here) and not two (i.e. Homebrew and python).
You can also install easy_install directly. The way to do this is install the distribute package using python3.3 explicitly.
Python 3.4 will make this much easier as pip will always be available