Python virtualenv: Why is pip still looking in a non-$path path? - python

I've been keeping all my python2.7 installs in my ~/.local/ directory that way I don't have to sudo every time I want to do a pip install. I also have $HOME/.local/lib/python2.7/site-packages on my $PYTHONPATH. This has worked well for years but now I find myself needing to run python3 programs more frequently. After much research it seems like virtualenv is the most recommended way to deal with python 2 and 3 on the same system. But I am running into troubles. I can spin up a python3 virtual environment but when I try to install new libs with pip, my old global path (i.e. ~/.local/) is still being searched by pip, which makes sense. However, this is even the case if I remove my ~/.local/bin/ directory from my $PATH and unset my $PYTHONPATH.
Here is are the steps I took:
First check the preliminaries before activating virtualenv. (I'm on Ubuntu 16.04 btw)
maddoxw#firefly:~$ echo $PATH
/usr/local/cuda-8.0/bin:/home/maddoxw/.node_modules_global/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/maddoxw/bin:/home/maddoxw/scripts
maddoxw#firefly:~$ echo $PYTHONPATH
maddoxw#firefly:~$ python --version
Python 2.7.12
maddoxw#firefly:~$ python3 --version
Python 3.5.2
maddoxw#firefly:~$ which pip
Since I removed my ~/.local/bin directory from my path, then I can be certain pip will not be found. Also, $PYTHONPATH is still empty. Now I create my virtualenv:
maddoxw#firefly:~$ mkdir test && cd test/
mkdir: created directory 'test'
maddoxw#firefly:~/test$ python3 -m venv .env
maddoxw#firefly:~/test$ source .env/bin/activate
(.env) maddoxw#firefly:~/test$ echo $PATH
/home/maddoxw/test/.env/bin:/usr/local/cuda-8.0/bin:/home/maddoxw/.node_modules_global/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/maddoxw/bin:/home/maddoxw/scripts
(.env) maddoxw#firefly:~/test$ echo $PYTHONPATH
(.env) maddoxw#firefly:~/test$ which python
/home/maddoxw/test/.env/bin/python
(.env) maddoxw#firefly:~/test$ python --version
Python 3.5.2
(.env) maddoxw#firefly:~/test$ which pip
/home/maddoxw/test/.env/bin/pip
Good. My ~/.local/ is still NOT on my $PATH, $PYTHONPATH is still empty, python points to the correct path and version, and pip is pointing to the correct location. Now lets try to pip install a fresh lib.
(.env) maddoxw#firefly:~/test$ pip install Cython
Requirement already satisfied: Cython in /home/maddoxw/.local/lib/python2.7/site-packages
Why is pip still looking in a non-$PATH path?

First, install pip3 to use with python3. You can install it with the following command and then use pip3 to install your packages.
sudo apt-get install python3-pip

[Solved]
When I initially set up my python2.7 environment way back when, I created for myself a handy little function wrapper around pip so that I wouldn't have to type out the --user every time I wanted to pip install
pip() {
if [ "$1" = "install" -o "$1" = "bundle" ]; then
cmd="$1"
shift
$HOME/.local/bin/pip $cmd --user $#
else
$HOME/.local/bin/pip $#
fi
}
I put this function in ~/.bash.d/bash_functions
and in my ~/.bashrc i added the line,
[ -f ~/.bash.d/bash_functions ] && source ~/.bash.d/bash_functions
So, although I removed $HOME/.local/ from my path. This wrapper function was still being called everytime I fired up a new terminal. Weather or not I was or was not in a virtualenv was irrelevant.
Solution?
Commented out (or delete completely) the function wrapper fixed it.

Related

How to install sqlite3 for python3.7 in seperate directory on linux without sudo commands?

I have the problem that when I run my code on a linux server I get:
ModuleNotFoundError: No module named '_sqlite3'
So after researching, I found out sqlite3 was supposed to have been installed when I installed python, however it didn't.
I think the problem comes from the way I installed python. Since I do not have sudo permissions, I installed python3.7 in a local directory using: This guide.
All solutions to this sqlite3 problem that I can find requires sudo commands.
Is there another way that I can install python3.7 together with sqlite3 in my local Linux directory without using any sudo commands?
I hope I have stated my question clearly and I would appreciate all the help I can get. Thank you!
While installing a python package in a Linux system without "sudo" privileges you can use
For Python 3
pip3 install --user pysqlite3
You can install any third party packages with the same method
pip3 install --user PACKAGE_NAME
The --user flag to pip install tells Pip to install packages in some specific directories within your home directory. For more information click here.
Hope it helps !
The solution is to first build sqlite3 into a user directory and then build python using that directory's libraries and include headers. In particular, #Ski has answered a similar question regarding python 2, which can be adopted to python 3:
$ mkdir -p ~/applications/src
$ cd ~/applications/src
$ # Download and build sqlite 3 (you might want to get a newer version)
$ wget http://www.sqlite.org/sqlite-autoconf-3070900.tar.gz
$ tar xvvf sqlite-autoconf-3070900.tar.gz
$ cd sqlite-autoconf-3070900
$ ./configure --prefix=~/applications
$ make
$ make install
$ # Now download and build python 2, same works for python 3
$ cd ~/applications/src
$ wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz
$ tar xvvf Python-2.5.2.tgz
$ cd Python-2.5.2
$ ./configure --prefix=~/applications
$ make
$ make install
$ ~/applications/bin/python
Alternatively, if you already have to specify a different --prefix for some reason (this has happened to me with pyenv), use LDFLAGS and CPPFLAGS when configuring python build:
$ ./configure LDFLAGS=-L/home/user/applications/lib/ CPPFLAGS=-I/home/user/applications/include/

pyvenv returns non-zero exit status 1 (during the installation of pip stage)

If you should ever encounter the following error when creating a Python virtual environment using the pyvenv command:
user$ pyvenv my_venv_dir
Error: Command '['/home/user/my_venv_dir/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1
... then the answer (below) provides a simple way to work around it, without resorting to setuptools and it's related acrobatics.
Here's an approach that is fairly O/S agnostic...
Both the pyvenv and python commands themselves include a --without-pip option that enable you to work around this issue; without resorting to setuptool or other headaches. Taking note of my inline comments below, here's how to do it, and is very easy to understand:
user$ pyvenv --without-pip ./pyvenv.d # Create virtual environment this way;
user$ python -m venv --without-pip ./pyvenv.d # --OR-- this newer way. Both work.
user$ source ./pyvenv.d/bin/activate # Now activate this new virtual environment.
(pyvenv.d) user$
# Within it, invoke this well-known script to manually install pip(1) into /pyvenv.d:
(pyvenv.d) user$ curl https://bootstrap.pypa.io/get-pip.py | python
(pyvenv.d) user$ deactivate # Next, reactivate this virtual environment,
user$ source ./pyvenv.d/bin/activate # which will now include the pip(1) command.
(pyvenv.d) user$
(pyvenv.d) user$ which pip # Verify that pip(1) is indeed present.
/path/to/pyvenv.d/bin/pip
(pyvenv.d) user$ pip install --upgrade pip # And finally, upgrade pip(1) itself;
(pyvenv.d) user$ # although it will likely be the
# latest version already.
# And that's it!
I hope this helps. \(◠﹏◠)/
2020, on python3.8 on WSL2 (Ubuntu) the following solved this for me:
sudo apt install python3.8-venv
If you have a module whose file is named after a python standard library, in the folder on which you invoke python -m venv venv, then this command will fail without a hint about that.
For instance, you name a file email.py.
What I did to find this was coding a bash script which moves the .py files off the current directory one by one (to a holdspace/ subdir), and try at each move to invoke the venv dir creation. If the python -m venv venv command exits with 0 code, so it's successful and the last file moved was the culprit.
#!/bin/bash
if [ ! -d ./holdspace ]
then
mkdir holdspace/
fi
for file in *.py
do
mv "$file" holdspace/
python -m venv venv >/dev/null 2>&1
if [ $? -eq 0 ]
then
echo "$file was the culprit."
rm -rf venv/
break
else
echo "$file isn't the culprit."
fi
rm -rf venv/
done
mv holdspace/*.py .
The following should fix it
brew update
brew upgrade
brew install zlib

Installing pip on Bluehost

I have successfully installed python 2.7.11 on a shared Bluehost server.
In the home directory I installed get-pip.py When I run that now,
# python get-pip.py
Requirement already up-to-date: pip in ./python/lib/python2.7/site-packages
But when I try to run pip I get,
usernmame#example.com [~]# pip
-bash: pip: command not found
Why is pip not running? How can I check what python packages are installed?
My ~/.bashrc looks like this,
# .bashrc
export PATH=$HOME/python/Python-2.7.11/:$PATH
# User specific aliases and functions
alias mv='mv -i'
alias rm='rm -i'
alias cp='cp -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
Additionally,
# echo $PATH
/usr/local/jdk/bin:/home2/username/python/Python-2.7.11/:/usr/lib64/qt-3.3/bin:/home2/username/perl5/bin:/ramdisk/php/54/bin:/usr/php/54/usr/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11R6/bin:/home2/username/bin
EDIT:
My attempt to add the path to $PATH in ~/.bashrc
export PATH=$HOME/python/Python-2.7.11/:$HOME/python/lib/python2.7/site-packages/:$PATH
But it doesn't work, same error.
Thanks
You can run pip by invoking the -m option when you use python to run pip. Suppose you want to use pip to install a Python package called "package-of-interest" from PyPI. Then, you can install it by running:
python -m pip install package-of-interest

Installing distribute in Python 3.3 venv (OS X/Homebrew)

I've been trying to get up and running with the built-in "venv" module of Python 3.3 on my OS X machine. I've installed Python 3.3 using Homebrew.
As per the docs, creating and switching virtual environment works as you'd expect:
$ python3 -m venv myvenv
$ source myvenv/bin/activate
And I've tested something like this:
$ echo "YEAH = 'YEAH!'" > myvenv/lib/python3.3/site-packages/thingy.py
$ python
>>> import thingy
>>> print(thingy.YEAH)
'YEAH!'
But when I try to install distribute, it simply won't go in the proper place. For some reason, it insists on trying to install into /usr/local/lib/python3.3/site-packages/, which fails with the following messages:
No setuptools distribution found
running install
Checking .pth file support in /usr/local/lib/python3.3/site-packages/
/Users/victor/myvenv/bin/python -E -c pass
TEST FAILED: /usr/local/lib/python3.3/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python3.3/site-packages/
and your PYTHONPATH environment variable currently contains:
''
This happens regardless if I try to install using distribute_setup.py or using the source distribution directly. I've even tried using --prefix=/Users/victor/myenv but it still tries to put everything in my "global" site-packages.
I can't figure out why this happens, but it's consistent on two of my machines. Note that sys.prefix reports the correct path (the virtual environment).
Is this a problem with Homebrew? OS X? Python 3.3? venv? Me?
This has been an issue with Homebrew, yes, but it is working now since https://github.com/mxcl/homebrew/commit/0b50110107ea2998e65011ec31ce45931b446dab.
$ brew update
$ brew rm python3 #if you have installed it before
$ brew install python3
$ cd /tmp
$ which python3
/usr/local/bin/python3
$ python3 -m venv myvenv
$ source myvenv/bin/activate
$ wget http://python-distribute.org/distribute_setup.py # may need brew install wget
$ python3 distribute_setup.py
...
Finished processing dependencies for distribute==0.6.45
After install bootstrap.
Creating /private/tmp/myvenv/lib/python3.3/site-packages/setuptools-0.6c11-py3.3.egg-info
Creating /private/tmp/myvenv/lib/python3.3/site-packages/setuptools.pth
You see that distribute install successfully into the /tmp dir.
This happens because homebrew installs distutils config file:
$ brew cat python3 | grep "Tell distutils" -A5
# Tell distutils-based installers where to put scripts
(prefix/"Frameworks/Python.framework/Versions/#{VER}/lib/python#{VER}/distutils/distutils.cfg").write <<-EOF.undent
[install]
install-scripts=#{scripts_folder}
install-lib=#{site_packages}
EOF
$ mv ~/.local/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/distutils.cfg ~/tmp/
$ cat ~/tmp/distutils.cfg
[install]
install-scripts=/Users/gatto/.local/share/python3
install-lib=/Users/gatto/.local/lib/python3.3/site-packages
$ . venv/bin/activate
(venv) $ python distribute-0.6.36/distribute_setup.py
(venv) $ ls venv/lib/python3.3/site-packages/
distribute-0.6.36-py3.3.egg easy-install.pth setuptools-0.6c11-py3.3.egg-info setuptools.pth
See "distutils.cfg Can Break venv" issue at bugs.python.org.

Is it possible to install another version of Python to Virtualenv?

I have a shared account in a web-hosting that has Python 2.4 installed, but my code is not compatible with 2.4. Is it possible to install Python 2.6 directly to Virtualenv?
Note: I don´t have permission to install it in the shared server.
Here are the options for virtualenv
$ virtualenv
You must provide a DEST_DIR
Usage: virtualenv [OPTIONS] DEST_DIR
Options:
--version show program's version number and exit.
-h, --help show this help message and exit.
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
-p PYTHON_EXE, --python=PYTHON_EXE
The Python interpreter to use, e.g.,
--python=python2.5 will use the python2.5 interpreter
to create the new environment. The default is the
interpreter that virtualenv was installed with
(/usr/bin/python)
--clear Clear out the non-root install and start from scratch
--no-site-packages Don't give access to the global site-packages dir to
the virtual environment
--unzip-setuptools Unzip Setuptools or Distribute when installing it
--relocatable Make an EXISTING virtualenv environment relocatable.
This fixes up scripts and makes all .pth files
relative
--distribute Use Distribute instead of Setuptools. Set environ
variable VIRTUALENV_USE_DISTRIBUTE to make it the
default
--prompt==PROMPT Provides an alternative prompt prefix for this
environment
1) What you want to do is install python to a directory that you are able to write too.
You can follow the instructions here.
For Python 2.7.1
Python source
mkdir ~/src
mkdir ~/.localpython
cd ~/src
wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz
tar -zxvf Python-2.7.1.tgz
cd Python-2.7.1
make clean
./configure --prefix=/home/${USER}/.localpython
make
make install
2) Install virtualenv
virtualenv source
cd ~/src
wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.5.2.tar.gz#md5=fbcefbd8520bb64bc24a560c6019a73c
tar -zxvf virtualenv-1.5.2.tar.gz
cd virtualenv-1.5.2/
~/.localpython/bin/python setup.py install
3) Create a virtualenv using your local python
virtualenv docs
mkdir /home/${USER}/virtualenvs
cd /home/${USER}/virtualenvs
~/.localpython/bin/virtualenv py2.7 --python=/home/${USER}/.localpython/bin/python2.7
4) Activate the environment
cd ~/virtualenvs/py2.7/bin
source ./activate
5) Check
(py2.7)$ python
Python 2.7.1 (r271:86832, Mar 31 2011, 15:31:37)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
(py2.7)$ deactivate
$ python
Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Pre-requisites:
sudo easy_install virtualenv
sudo pip install virtualenvwrapper
Installing virtualenv with Python2.6:
You could manually download, build and install another version of Python to /usr/local or another location.
If it's another location other than /usr/local, add it to your PATH.
Reload your shell to pick up the updated PATH.
From this point on, you should be able to call the following 2 python binaries from your shell python2.5 and python2.6
Create a new instance of virtualenv with python2.6:
mkvirtualenv --python=python2.6 yournewenv
Now a days, the easiest way I found to have a more updated version of Python is to install it via conda into a conda environment.
Install conda(you may need a virtualenv for this)
pip install conda
Installing a new Python version inside a conda environent
I'm adding this answer here because no manual download is needed. conda will do that for you.
Now create an environment for the Python version you want. In this example I will use 3.5.2, because it it the latest version at this time of writing (Aug 2016).
conda create -n py35 python=3.5.2
Will create a environment for conda to install packages
To activate this environment(I'm assuming linux otherwise check the conda docs):
source activate py35
Now install what you need either via pip or conda in the environemnt(conda has better binary package support).
conda install <package_name>
Full guide with pyenv
If pyenv is not installed then install it with pyenv-installer:
$ curl https://pyenv.run | bash
To use any custom python version, e.g. 3.5.6 use the following:
pyenv install 3.5.6
pyenv virtualenv 3.5.6 NAME_OF_YOUR_ENV
cd YOUR_PROJECT_PATH
pyenv local NAME_OF_YOUR_ENV
The usual approach is to download the source and build and install locally (but not directly in virtualenv), and then create a new virtualenv using that local Python install. On some systems, it may be possible to download and install a prebuilt python, rather than building from source.
This procedure installs Python2.7 anywhere and eliminates any absolute path references within your env folder (managed by virtualenv). Even virtualenv isn't installed absolutely.
Thus, theoretically, you can drop the top level directory into a tarball, distribute, and run anything configured within the tarball on a machine that doesn't have Python (or any dependencies) installed.
Contact me with any questions. This is just part of an ongoing, larger project I am engineering. Now, for the drop...
Set up environment folders.
$ mkdir env
$ mkdir pyenv
$ mkdir dep
Get Python-2.7.3, and virtualenv without any form of root OS installation.
$ cd dep
$ wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
$ wget https://raw.github.com/pypa/virtualenv/master/virtualenv.py
Extract and install Python-2.7.3 into the pyenv dir. make clean is optional if you are doing this a 2nd, 3rd, Nth time...
$ tar -xzvf Python-2.7.3.tgz
$ cd Python-2.7.3
$ make clean
$ ./configure --prefix=/path/to/pyenv
$ make && make install
$ cd ../../
$ ls
dep env pyenv
Create your virtualenv
$ dep/virtualenv.py --python=/path/to/pyenv/bin/python --verbose env
Fix the symlink to python2.7 within env/include/
$ ls -l env/include/
$ cd !$
$ rm python2.7
$ ln -s ../../pyenv/include/python2.7 python2.7
$ cd ../../
Fix the remaining python symlinks in env. You'll have to delete the symbolically linked directories and recreate them, as above. Also, here's the syntax to force in-place symbolic link creation.
$ ls -l env/lib/python2.7/
$ cd !$
$ ln -sf ../../../pyenv/lib/python2.7/UserDict.py UserDict.py
[...repeat until all symbolic links are relative...]
$ cd ../../../
Test
$ python --version
Python 2.7.1
$ source env/bin/activate
(env)
$ python --version
Python 2.7.3
Aloha.
I'm using virtualenvwrapper and don't want to modify $PATH, here's how:
$ which python3
/usr/local/bin/python3
$ mkvirtualenv --python=/usr/local/bin/python3 env_name
You may use pyenv.
There are a lot of different versions anaconda, jython, pypy and so on...
https://github.com/yyuu/pyenv
Installation as simple as pyenv install 3.2.6
pyenv install --list
Available versions:
2.1.3
2.2.3
2.3.7
2.4
2.4.1
2.4.2
2.4.3
2.4.4
2.4.5
2.4.6
2.5
2.5.1
2.5.2
2.5.3
2.5.4
2.5.5
2.5.6
2.6.6
...
Although the question specifically describes installing 2.6, I would like to add some importants points to the excellent answers above in case someone comes across this. For the record, my case was that I was trying to install 2.7 on an ubuntu 10.04 box.
First, my motivation towards the methods described in all the answers here is that installing Python from deadsnake's ppa's has been a total failure. So building a local Python is the way to go.
Having tried so, I thought relying to the default installation of pip (with sudo apt-get install pip) would be adequate. This unfortunately is wrong. It turned out that I was getting all shorts of nasty issues and eventually not being able to create a virtualenv.
Therefore, I highly recommend to install pip locally with wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py && python get-pip.py --user. This related question gave me this hint.
Now if this doesn't work, make sure that libssl-dev for Ubuntu or openssl-dev for CentOS is installed. Install them with apt-get or yum and then re-build Python (no need to remove anything if already installed, do so on top). get-pip complains about that, you can check so by running import ssl on a py shell.
Last, don't forget to declare .local/bin and local python to path, check with which pip and which python.
No, but you can install an isolated Python build (such as ActivePython) under your $HOME directory.
This approach is the fastest, and doesn't require you to compile Python yourself.
(as a bonus, you also get to use ActiveState's binary package manager)
I have not found suitable answer, so here goes my take, which builds upon #toszter answer, but does not use system Python (and you may know, it is not always good idea to install setuptools and virtualenv at system level when dealing with many Python configurations):
#!/bin/sh
mkdir python_ve
cd python_ve
MYROOT=`pwd`
mkdir env pyenv dep
cd ${MYROOT}/dep
wget https://pypi.python.org/packages/source/s/setuptools/setuptools-15.2.tar.gz#md5=a9028a9794fc7ae02320d32e2d7e12ee
wget https://raw.github.com/pypa/virtualenv/master/virtualenv.py
wget https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tar.xz
xz -d Python-2.7.9.tar.xz
cd ${MYROOT}/pyenv
tar xf ../dep/Python-2.7.9.tar
cd Python-2.7.9
./configure --prefix=${MYROOT}/pyenv && make -j 4 && make install
cd ${MYROOT}/pyenv
tar xzf ../dep/setuptools-15.2.tar.gz
cd ${MYROOT}
pyenv/bin/python dep/virtualenv.py --no-setuptools --python=${MYROOT}/pyenv/bin/python --verbose env
env/bin/python pyenv/setuptools-15.2/setup.py install
env/bin/easy_install pip
echo "virtualenv in ${MYROOT}/env"
The trick of breaking chicken-egg problem here is to make virtualenv without setuptools first, because it otherwise fails (pip can not be found). It may be possible to install pip / wheel directly, but somehow easy_install was the first thing which came to my mind. Also, the script can be improved by factoring out concrete versions.
NB. Using xz in the script.
First of all, Thank you DTing for awesome answer. It's pretty much perfect.
For those who are suffering from not having GCC access in shared hosting, Go for ActivePython instead of normal python like Scott Stafford mentioned. Here are the commands for that.
wget http://downloads.activestate.com/ActivePython/releases/2.7.13.2713/ActivePython-2.7.13.2713-linux-x86_64-glibc-2.3.6-401785.tar.gz
tar -zxvf ActivePython-2.7.13.2713-linux-x86_64-glibc-2.3.6-401785.tar.gz
cd ActivePython-2.7.13.2713-linux-x86_64-glibc-2.3.6-401785
./install.sh
It will ask you path to python directory. Enter
../../.localpython
Just replace above as Step 1 in DTing's answer and go ahead with Step 2 after that. Please note that ActivePython package URL may change with new release. You can always get new URL from here : http://www.activestate.com/activepython/downloads
Based on URL you need to change the name of tar and cd command based on file received.
virtualenv --python=".localpython/bin/python2.7" env

Categories

Resources