Introduction
I would like to install some dependencies (local-package1 and local-package2) into a virtualenv using Tox. Those dependencies are packages that only exist on my local Python Package Index (inside of an Artifactory instance).
Attempts
Here's my tox.ini:
[tox]
envlist = py27
indexserver =
LOCAL = https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple
[testenv]
deps =
:LOCAL:local-package1
mock
pykwalify
:LOCAL:local-package2
xlrd
XlsxWriter
xlwt
yamllint
commands =
make.sh
Here's the output from tox -v:
using tox.ini: repo/tox.ini
using tox-3.7.0 from /usr/local/lib/python2.7/site-packages/tox/__init__.pyc
GLOB sdist-make: ~/repo/setup.py
~/repo$ /usr/local/opt/python#2/bin/python2.7 ~/repo/setup.py sdist --formats=zip --dist-dir ~/repo/.tox/dist >~/repo/.tox/log/tox-0.log
package .tmp/package/1/example-0.0.0.zip links to dist/example-0.0.0.zip (~/repo/.tox)
py27 cannot reuse: no previous config ~/repo/.tox/py27/.tox-config1
py27 create: ~/repo/.tox/py27
~/repo/.tox$ /usr/local/opt/python#2/bin/python2.7 -m virtualenv --python /usr/local/opt/python#2/bin/python2.7 py27 >~/repo/.tox/py27/log/py27-0.log
py27 installdeps: :LOCAL:local-package1, mock, pykwalify, :LOCAL:local-package2, xlrd, XlsxWriter, xlwt, yamllint
~/repo$ ~/repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2 >~/repo/.tox/py27/log/py27-1.log
And doing a tail -f ~/repo/.tox/py27/log/py27-1.log yields the following:
actionid: py27
msg: getenv
cmdargs: 'repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2'
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: https://me:****#artifactory.example.com/api/pypi/pypi-local/simple, https://artifactory.example.com/api/pypi/pypi-local/simple
Collecting local-package1
and it seems to hang indefinitely.
If I run the command:
repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2
outside of the virtualenv, I can install these packages without any issues.
Question
How can I properly install dependencies with Tox which includes authentication to a Python Package Index?
Related
I am trying to simplify the console output of tox and to hide the list of installed packages, ideally to hide them if there are no errors installing them.
Here is the current output
tox -e py27
py27 installed: alabaster==0.7.9,...<VERY LONG LIST..
py27 runtests: ...
I tried to add install_command = pip install -q {opts} {packages} but it did had no effect.
With tox >= 2.4 you can use the list_dependencies_command parameter (http://tox.readthedocs.io/en/latest/config.html#confval-list_dependencies_command).
[testenv]
whitelist_externals=echo
list_dependencies_command=echo
It seems that at this moment this is a tox bug https://github.com/tox-dev/tox/issues/601
Introduction
I'm just starting to use AWS Lambda and as much as I hate it, I freaking love it. I've created a Makefile to help me package my virtual env and ship to S3. After I figured out that cryptography requires a hidden file in the site-packages directory #GRRR, I started wondering how I can further improve my packaging process.
Context
This is what a new virtualenv on a new Amazon Linux AMI EC2 instance looks like.
$ uname -srvm
Linux 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64
$ cat /etc/system-release
Amazon Linux AMI release 2016.09
$ virtualenv --version
15.1.0
$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
$ virtualenv temp
New python executable in /home/ec2-user/temp/bin/python2.7
Also creating executable in /home/ec2-user/temp/bin/python
Installing setuptools, pip, wheel...done.
fig. 1
$ ls -a temp/lib/python2.7/site-packages/
. packaging-16.8.dist-info setuptools-34.3.2.dist-info
.. pip six-1.10.0.dist-info
appdirs-1.4.3.dist-info pip-9.0.1.dist-info six.py
appdirs.py pkg_resources six.pyc
appdirs.pyc pyparsing-2.2.0.dist-info wheel
easy_install.py pyparsing.py wheel-0.29.0.dist-info
easy_install.pyc pyparsing.pyc
packaging setuptools
fig. 2
I found that in order to do the python development I needed (using paramiko), I had to do this to prepare (prior to fig.1 & fig.2):
sudo yum install gcc python27-devel libffi-devel openssl-devel
sudo -H pip install --upgrade pip virtualenv
fig. 3
Question
Of those site-packages in fig. 2, which ones can I omit from the zip I send to AWS?
For the sake of comparison, this is what my complete project's virtualenv has in it (and the only thing I pip installed was paramiko):
$ ls -a aws_lambda_project/lib/python2.7/site-packages/
. packaging
.. packaging-16.8.dist-info
appdirs-1.4.3.dist-info paramiko
appdirs.py paramiko-2.1.2.dist-info
appdirs.pyc pip
asn1crypto pip-9.0.1.dist-info
asn1crypto-0.22.0.dist-info pkg_resources
cffi pyasn1
cffi-1.9.1.dist-info pyasn1-0.2.3.dist-info
_cffi_backend.so pycparser
cryptography pycparser-2.17.dist-info
cryptography-1.8.1.dist-info pyparsing-2.2.0.dist-info
easy_install.py pyparsing.py
easy_install.pyc pyparsing.pyc
enum setuptools
enum34-1.1.6.dist-info setuptools-34.3.2.dist-info
idna six-1.10.0.dist-info
idna-2.5.dist-info six.py
ipaddress-1.0.18.dist-info six.pyc
ipaddress.py wheel
ipaddress.pyc wheel-0.29.0.dist-info
.libs_cffi_backend
This works for me, please give it a try:
$ mkdir paramiko-lambda && cd paramiko-lambda
$ virtualenv env --python=python2.7 && source env/bin/activate
$ pip freeze > pre_paramiko.txt
$ pip install paramiko
$ pip freeze > post_paramiko.txt
I then put the following in a script to make sure it works locally:
from __future__ import print_function
import paramiko
def handler(event, context):
print(paramiko.__version__)
ssh_client = paramiko.SSHClient()
if __name__ == '__main__':
handler(event=None, context=None)
The last two lines are optional, just a simple way to test the script locally. To see what was installed along with paramiko, I compared the two text files:
$ diff -u pre_paramiko.txt post_paramiko.txt
--- pre_paramiko.txt
+++ post_paramiko.txt
## -1,4 +1,13 ##
appdirs==1.4.3
+asn1crypto==0.22.0
+cffi==1.10.0
+cryptography==1.8.1
+enum34==1.1.6
+idna==2.5
+ipaddress==1.0.18
packaging==16.8
+paramiko==2.1.2
+pyasn1==0.2.3
+pycparser==2.17
pyparsing==2.2.0
six==1.10.0
The modules with a + were installed with paramiko so must be included with .zip archive that gets uploaded to AWS Lambda. It would be easy to write a bash script that takes the output of the diff command and automate the creation of the .zip archive, but I'm just going to enter them in manually.
$ cd env/lib/python2.7/site-packages
$ zip -x "*.pyc" -r ../../../../paramiko_lambda.zip packaging asn1crypto cffi cryptography enum idna ipaddress paramiko pyasn1 pycparser
$ cd ../../../../
$ zip -r paramiko_lambda.zip paramiko_lambda.py
I needed to add the packaging folder probably because of print(paramiko.__version__) so it may not be necessary. The paramiko_lambda.zip file was 2.5 MB and while not huge had a lot of unnecessary data, specifically *.pyc files. Excluding *.pyc files reduced the file to 1.5 MB.
I have successfully installed python 2.7.11 on a shared Bluehost server.
In the home directory I installed get-pip.py When I run that now,
# python get-pip.py
Requirement already up-to-date: pip in ./python/lib/python2.7/site-packages
But when I try to run pip I get,
usernmame#example.com [~]# pip
-bash: pip: command not found
Why is pip not running? How can I check what python packages are installed?
My ~/.bashrc looks like this,
# .bashrc
export PATH=$HOME/python/Python-2.7.11/:$PATH
# User specific aliases and functions
alias mv='mv -i'
alias rm='rm -i'
alias cp='cp -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
Additionally,
# echo $PATH
/usr/local/jdk/bin:/home2/username/python/Python-2.7.11/:/usr/lib64/qt-3.3/bin:/home2/username/perl5/bin:/ramdisk/php/54/bin:/usr/php/54/usr/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11R6/bin:/home2/username/bin
EDIT:
My attempt to add the path to $PATH in ~/.bashrc
export PATH=$HOME/python/Python-2.7.11/:$HOME/python/lib/python2.7/site-packages/:$PATH
But it doesn't work, same error.
Thanks
You can run pip by invoking the -m option when you use python to run pip. Suppose you want to use pip to install a Python package called "package-of-interest" from PyPI. Then, you can install it by running:
python -m pip install package-of-interest
I have a Python package configured like this:
# setup.py
from setuptools import setup
setup(
name='python-package-test',
version='0.0.1',
packages=['python-package-test'],
dependency_links=[
# This repo actually exists
'git+https://github.com/nhooey/tendo.git#increase-version-0.2.9#egg=tendo',
],
install_requires=[
'tendo',
],
)
When I install this package from setup.py:
$ virtualenv --python=python3 .venv && \
source .venv/bin/activate && \
python setup.py install
$ pip freeze | grep tendo
tendo==0.2.9 # Note that this is the *correct* version
It installs the correct version of tendo.
However, when I upload this package in a Git repository and install it with pip:
# The GitHub link doesn't exist as it's private
# and it's different from the repo mentioned above
virtualenv --python=python3 .venv && \
source .venv/bin/activate && \
pip install git+ssh://git#github.com/nhooey/package.git
$ pip freeze | grep tendo
tendo==0.2.8 # Note that this is the *wrong* version
It installs the wrong version of tendo.
Why is the setup.py installation behaving differently from pip + git?
You must use the --process-dependency-links option when installing with Pip, since Pip no longer processes this automatically.
pip install --process-dependency-links 'git+ssh://git#github.com/nhooey/package.git'
You'd think Pip would print a warning, or the updated version of setuptools would also ignore dependency_links as well.
I've been trying to get up and running with the built-in "venv" module of Python 3.3 on my OS X machine. I've installed Python 3.3 using Homebrew.
As per the docs, creating and switching virtual environment works as you'd expect:
$ python3 -m venv myvenv
$ source myvenv/bin/activate
And I've tested something like this:
$ echo "YEAH = 'YEAH!'" > myvenv/lib/python3.3/site-packages/thingy.py
$ python
>>> import thingy
>>> print(thingy.YEAH)
'YEAH!'
But when I try to install distribute, it simply won't go in the proper place. For some reason, it insists on trying to install into /usr/local/lib/python3.3/site-packages/, which fails with the following messages:
No setuptools distribution found
running install
Checking .pth file support in /usr/local/lib/python3.3/site-packages/
/Users/victor/myvenv/bin/python -E -c pass
TEST FAILED: /usr/local/lib/python3.3/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python3.3/site-packages/
and your PYTHONPATH environment variable currently contains:
''
This happens regardless if I try to install using distribute_setup.py or using the source distribution directly. I've even tried using --prefix=/Users/victor/myenv but it still tries to put everything in my "global" site-packages.
I can't figure out why this happens, but it's consistent on two of my machines. Note that sys.prefix reports the correct path (the virtual environment).
Is this a problem with Homebrew? OS X? Python 3.3? venv? Me?
This has been an issue with Homebrew, yes, but it is working now since https://github.com/mxcl/homebrew/commit/0b50110107ea2998e65011ec31ce45931b446dab.
$ brew update
$ brew rm python3 #if you have installed it before
$ brew install python3
$ cd /tmp
$ which python3
/usr/local/bin/python3
$ python3 -m venv myvenv
$ source myvenv/bin/activate
$ wget http://python-distribute.org/distribute_setup.py # may need brew install wget
$ python3 distribute_setup.py
...
Finished processing dependencies for distribute==0.6.45
After install bootstrap.
Creating /private/tmp/myvenv/lib/python3.3/site-packages/setuptools-0.6c11-py3.3.egg-info
Creating /private/tmp/myvenv/lib/python3.3/site-packages/setuptools.pth
You see that distribute install successfully into the /tmp dir.
This happens because homebrew installs distutils config file:
$ brew cat python3 | grep "Tell distutils" -A5
# Tell distutils-based installers where to put scripts
(prefix/"Frameworks/Python.framework/Versions/#{VER}/lib/python#{VER}/distutils/distutils.cfg").write <<-EOF.undent
[install]
install-scripts=#{scripts_folder}
install-lib=#{site_packages}
EOF
$ mv ~/.local/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/distutils.cfg ~/tmp/
$ cat ~/tmp/distutils.cfg
[install]
install-scripts=/Users/gatto/.local/share/python3
install-lib=/Users/gatto/.local/lib/python3.3/site-packages
$ . venv/bin/activate
(venv) $ python distribute-0.6.36/distribute_setup.py
(venv) $ ls venv/lib/python3.3/site-packages/
distribute-0.6.36-py3.3.egg easy-install.pth setuptools-0.6c11-py3.3.egg-info setuptools.pth
See "distutils.cfg Can Break venv" issue at bugs.python.org.