Develop Reference

How to install pip packages in container [duplicate]

This question already has answers here:
Activate python virtualenv in Dockerfile
(6 answers)
Closed 2 years ago.
I am trying to setup a python virtual environment on a docker image running a docker build
The terminal output is ok when I run docker build .. but when I login into my container, no packages are installer in my virtual environment.
#Dockerfile
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python3.8 get-pip.py
RUN pip install virtualenv
RUN virtualenv venv
RUN /home/ubuntu/venv/bin/pip install -r auto/requirements.txt
...
# Docker build command
docker build --no-cache -t auto-server:1.0 .
# Terminal output
Step 27/27 : RUN /home/ubuntu/venv/bin/pip install -r auto/requirements.txt
---> Running in d27dbb9a4c97
Collecting asgiref==3.2.10
Downloading asgiref-3.2.10-py3-none-any.whl (19 kB)
Collecting beautifulsoup4==4.9.1
Downloading beautifulsoup4-4.9.1-py3-none-any.whl (115 kB)
Collecting Django==3.1.1
Downloading Django-3.1.1-py3-none-any.whl (7.8 MB)
...
Successfully installed Django-3.1.1 asgiref-3.2.10 beautifulsoup4-4.9.1 fake-useragent-0.1.11 joblib-0.16.0 numpy-1.19.2 pandas-1.1.2 python-dateutil-2.8.1 pytz-2020.1 scikit-learn-0.23.2 scipy-1.5.2 six-1.15.0 sklearn-0.0 soupsieve-2.0.1 sqlparse-0.3.1 threadpoolctl-2.1.0
Here is what I get when I list pakages in my virtual environment:
$ docker exec -ti auto-server bash
root#9c1f914d1b7b:/home/ubuntu# source venv/bin/activate
(venv) root#9c1f914d1b7b:/home/ubuntu# pip list
Package Version
---------- -------
pip 20.2.2
setuptools 49.6.0
wheel 0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/home/ubuntu/venv/bin/python -m pip install --upgrade pip' command.
Is it the right way to do it? How to make sure packages will be installed?

Finally having a virtual environment is not mandatory in a container development environment, I just setup image python environment as desired.
# Dockerfile
...
RUN python3.8 get-pip.py
RUN pip install -r auto/requirements.txt
This is not exactly what I was looking for but it does the job.

I use simple Dockerfile for dev mode
# pull official base image
FROM python:3.6-alpine
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apk update \
&& apk add postgresql-dev gcc python3-dev
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# copy project
COPY . .

How to Install Dependencies from a Local Python Repository with Tox?

Introduction
I would like to install some dependencies (local-package1 and local-package2) into a virtualenv using Tox. Those dependencies are packages that only exist on my local Python Package Index (inside of an Artifactory instance).
Attempts
Here's my tox.ini:
[tox]
envlist = py27
indexserver =
LOCAL = https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple
[testenv]
deps =
:LOCAL:local-package1
mock
pykwalify
:LOCAL:local-package2
xlrd
XlsxWriter
xlwt
yamllint
commands =
make.sh
Here's the output from tox -v:
using tox.ini: repo/tox.ini
using tox-3.7.0 from /usr/local/lib/python2.7/site-packages/tox/__init__.pyc
GLOB sdist-make: ~/repo/setup.py
~/repo$ /usr/local/opt/python#2/bin/python2.7 ~/repo/setup.py sdist --formats=zip --dist-dir ~/repo/.tox/dist >~/repo/.tox/log/tox-0.log
package .tmp/package/1/example-0.0.0.zip links to dist/example-0.0.0.zip (~/repo/.tox)
py27 cannot reuse: no previous config ~/repo/.tox/py27/.tox-config1
py27 create: ~/repo/.tox/py27
~/repo/.tox$ /usr/local/opt/python#2/bin/python2.7 -m virtualenv --python /usr/local/opt/python#2/bin/python2.7 py27 >~/repo/.tox/py27/log/py27-0.log
py27 installdeps: :LOCAL:local-package1, mock, pykwalify, :LOCAL:local-package2, xlrd, XlsxWriter, xlwt, yamllint
~/repo$ ~/repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2 >~/repo/.tox/py27/log/py27-1.log
And doing a tail -f ~/repo/.tox/py27/log/py27-1.log yields the following:
actionid: py27
msg: getenv
cmdargs: 'repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2'
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: https://me:****#artifactory.example.com/api/pypi/pypi-local/simple, https://artifactory.example.com/api/pypi/pypi-local/simple
Collecting local-package1
and it seems to hang indefinitely.
If I run the command:
repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2
outside of the virtualenv, I can install these packages without any issues.
Question
How can I properly install dependencies with Tox which includes authentication to a Python Package Index?

When zipping a virtual env for AWS Lambda deployment, what can I leave out?

Introduction
I'm just starting to use AWS Lambda and as much as I hate it, I freaking love it. I've created a Makefile to help me package my virtual env and ship to S3. After I figured out that cryptography requires a hidden file in the site-packages directory #GRRR, I started wondering how I can further improve my packaging process.
Context
This is what a new virtualenv on a new Amazon Linux AMI EC2 instance looks like.
$ uname -srvm
Linux 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64
$ cat /etc/system-release
Amazon Linux AMI release 2016.09
$ virtualenv --version
15.1.0
$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
$ virtualenv temp
New python executable in /home/ec2-user/temp/bin/python2.7
Also creating executable in /home/ec2-user/temp/bin/python
Installing setuptools, pip, wheel...done.
fig. 1
$ ls -a temp/lib/python2.7/site-packages/
. packaging-16.8.dist-info setuptools-34.3.2.dist-info
.. pip six-1.10.0.dist-info
appdirs-1.4.3.dist-info pip-9.0.1.dist-info six.py
appdirs.py pkg_resources six.pyc
appdirs.pyc pyparsing-2.2.0.dist-info wheel
easy_install.py pyparsing.py wheel-0.29.0.dist-info
easy_install.pyc pyparsing.pyc
packaging setuptools
fig. 2
I found that in order to do the python development I needed (using paramiko), I had to do this to prepare (prior to fig.1 & fig.2):
sudo yum install gcc python27-devel libffi-devel openssl-devel
sudo -H pip install --upgrade pip virtualenv
fig. 3
Question
Of those site-packages in fig. 2, which ones can I omit from the zip I send to AWS?
For the sake of comparison, this is what my complete project's virtualenv has in it (and the only thing I pip installed was paramiko):
$ ls -a aws_lambda_project/lib/python2.7/site-packages/
. packaging
.. packaging-16.8.dist-info
appdirs-1.4.3.dist-info paramiko
appdirs.py paramiko-2.1.2.dist-info
appdirs.pyc pip
asn1crypto pip-9.0.1.dist-info
asn1crypto-0.22.0.dist-info pkg_resources
cffi pyasn1
cffi-1.9.1.dist-info pyasn1-0.2.3.dist-info
_cffi_backend.so pycparser
cryptography pycparser-2.17.dist-info
cryptography-1.8.1.dist-info pyparsing-2.2.0.dist-info
easy_install.py pyparsing.py
easy_install.pyc pyparsing.pyc
enum setuptools
enum34-1.1.6.dist-info setuptools-34.3.2.dist-info
idna six-1.10.0.dist-info
idna-2.5.dist-info six.py
ipaddress-1.0.18.dist-info six.pyc
ipaddress.py wheel
ipaddress.pyc wheel-0.29.0.dist-info
.libs_cffi_backend

This works for me, please give it a try:
$ mkdir paramiko-lambda && cd paramiko-lambda
$ virtualenv env --python=python2.7 && source env/bin/activate
$ pip freeze > pre_paramiko.txt
$ pip install paramiko
$ pip freeze > post_paramiko.txt
I then put the following in a script to make sure it works locally:
from __future__ import print_function
import paramiko
def handler(event, context):
print(paramiko.__version__)
ssh_client = paramiko.SSHClient()
if __name__ == '__main__':
handler(event=None, context=None)
The last two lines are optional, just a simple way to test the script locally. To see what was installed along with paramiko, I compared the two text files:
$ diff -u pre_paramiko.txt post_paramiko.txt
--- pre_paramiko.txt
+++ post_paramiko.txt
## -1,4 +1,13 ##
appdirs==1.4.3
+asn1crypto==0.22.0
+cffi==1.10.0
+cryptography==1.8.1
+enum34==1.1.6
+idna==2.5
+ipaddress==1.0.18
packaging==16.8
+paramiko==2.1.2
+pyasn1==0.2.3
+pycparser==2.17
pyparsing==2.2.0
six==1.10.0
The modules with a + were installed with paramiko so must be included with .zip archive that gets uploaded to AWS Lambda. It would be easy to write a bash script that takes the output of the diff command and automate the creation of the .zip archive, but I'm just going to enter them in manually.
$ cd env/lib/python2.7/site-packages
$ zip -x "*.pyc" -r ../../../../paramiko_lambda.zip packaging asn1crypto cffi cryptography enum idna ipaddress paramiko pyasn1 pycparser
$ cd ../../../../
$ zip -r paramiko_lambda.zip paramiko_lambda.py
I needed to add the packaging folder probably because of print(paramiko.__version__) so it may not be necessary. The paramiko_lambda.zip file was 2.5 MB and while not huge had a lot of unnecessary data, specifically *.pyc files. Excluding *.pyc files reduced the file to 1.5 MB.

How do I deploy a Python application to Amazon Elastic Beanstalk from Jenkins?

I am trying to deploy to Amazon Elastic Beanstalk programmatically from a Jenkins job. On my development machine, this is as simple as:
eb deploy $(AWS_ELASTIC_BEANSTALK_ENVIRONMENT)
On Jenkins, it should be as simple as configuring the following as a build command:
virtualenv env && source env/bin/activate && pip install awsebcli
mkdir -p .elasticbeanstalk
cat << EOF > .elasticbeanstalk/config.yml
branch-defaults:
master:
environment: myenv
global:
application_name: myapp
default_ec2_keyname: null
default_platform: 64bit Amazon Linux 2014.09 v1.0.9 running Python 2.7
default_region: us-west-2
profile: eb-cli
sc: git
EOF
eb deploy myenv
However, this fails with the following trace:
[EnvInject] - Loading node environment variables.
Building remotely on standard Amazon Linux 2014.09 AMI (i-d39710df) (x) in workspace /media/ephemeral0/jenkins/workspace/My_Job
Fetching changes from the remote Git repository
Fetching upstream changes from git#github.com:myapp.git
Checking out Revision be45db94111f9ab49fe8031eb583307d2fb9921c (origin/master)
[My_Job] $ /bin/sh -xe /tmp/hudson8633484437962332339.sh
+ virtualenv env
New python executable in env/bin/python2.7
Not overwriting existing python script env/bin/python (you must use env/bin/python2.7)
Installing Setuptools..................................................................................................................done.
Installing Pip....................................................................................................................................done.
+ source env/bin/activate
++ deactivate nondestructive
++ unset pydoc
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/media/ephemeral0/jenkins/workspace/My_Job/env
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/usr/local/bin:/bin:/usr/bin:/opt/aws/bin
++ PATH=/media/ephemeral0/jenkins/workspace/My_Job/env/bin:/usr/local/bin:/bin:/usr/bin:/opt/aws/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' x '!=' x ']'
+++ basename /media/ephemeral0/jenkins/workspace/My_Job/env
++ '[' env = __ ']'
+++ basename /media/ephemeral0/jenkins/workspace/My_Job/env
++ PS1='(env)'
++ export PS1
++ alias 'pydoc=python -m pydoc'
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
+ pip install awsebcli
Requirement already satisfied (use --upgrade to upgrade): awsebcli in ./env/lib/python2.7/site-packages
Downloading/unpacking setuptools>=7.0 (from awsebcli)
Running setup.py egg_info for package setuptools
Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.11 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): six==1.8.0 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): cement==2.4 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.2 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): jmespath>=0.4.1 in ./env/lib/python2.7/site-packages (from awsebcli)
Installing collected packages: setuptools
Found existing installation: setuptools 0.9.7
Uninstalling setuptools:
Successfully uninstalled setuptools
Running setup.py install for setuptools
Installing easy_install script to /media/ephemeral0/jenkins/workspace/My_Job/env/bin
Installing easy_install-2.7 script to /media/ephemeral0/jenkins/workspace/My_Job/env/bin
Successfully installed setuptools
Cleaning up...
+ mkdir -p .elasticbeanstalk
+ cat
+ cat .elasticbeanstalk/config.yml
branch-defaults:
master:
environment: myenv
global:
application_name: myapp
default_ec2_keyname: null
default_platform: 64bit Amazon Linux 2014.09 v1.0.9 running Python 2.7
default_region: us-west-2
profile: eb-cli
sc: git
+ eb deploy myenv
ERROR: The config profile (eb-cli) could not be found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
It's unclear why this happens since when I run the above on a local copy of my project it works fine.
The error message doesn't seem to be much help. It's unclear why eb-cli would be unable to found on the Jenkins machine.
So to summarize again my question: How do I deploy to Amazon Elastic Beanstalk from Jenkins? Is the above approach correct, but with errors in the details? Or is there some simpler way entirely?

To correct the config profile (eb-cli) could not be found error, drop the credentials you are using to deploy to EB into ~/.aws/config for your jenkins user on your jenkins machine. If you built your deployment on a local machine, you should be able to pull the file directly from ~/.aws/config locally. It will look like this:
[profile eb-cli]
aws_access_key_id = (for your IAM user)
aws_secret_access_key = (for your IAM user)

I resolved this by sshing into the Jenkins machine, running eb init, and then comparing the generated .elasticbeanstalk/config.yml with the one in the here-doc I was using above. The two were different because of the different security profiles on my development machine versus the Jenkins machine.
We can rewrite this script to be more robust against different config.yaml files like this:
virtualenv env && source env/bin/activate && pip install awsebcli
echo "1" | eb init myapp --region us-west-2 && eb use myenv && eb deploy myenv
Note, we use echo "1" | eb init myapp --region us-west-2 to select a default environment since eb init does not take environment as a positional argument and then use eb use myenv to select the environment we want.

We ran into this issue out of the blue, this may or may not be relevant, but I wanted to get this out there for anyone that might run across this again. The eb-cli seems to have changed slightly and will not allow credentials to be set globally.
Our JenkinsFile looks like this
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: 'iam-creds']]) {
sh "pip install awsebcli --upgrade --user"
sh "~/.local/bin/eb use my-application"
sh "~/.local/bin/eb deploy --verbose"
}
and our config.yml looks like this
global:
application_name: my-application
default_ec2_keyname: application-keyname
...
profile: eb-cli
sc: git
workspace_type: Application
removing the profile key solves the issue... However, this disallows deployments from the local machine (unless there is a way to use globals)

Pytest doesn't recognize -n option after pytest-xdist installation

I have installed pytest-xdist on top of a working pytest environment :
pip install pytest-xdist
and I have received this output
Downloading/unpacking pytest-xdist
Downloading pytest-xdist-1.10.tar.gz
Running setup.py egg_info for package pytest-xdist
no previously-included directories found matching '.hg'
Downloading/unpacking execnet>=1.1 (from pytest-xdist)
Downloading execnet-1.2.0.tar.gz (163kB): 163kB downloaded
Running setup.py egg_info for package execnet
warning: no files found matching 'conftest.py'
Requirement already satisfied (use --upgrade to upgrade): pytest>=2.4.2 in /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages (from pytest-xdist)
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.20 in /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages (from pytest>=2.4.2->pytest-xdist)
Installing collected packages: pytest-xdist, execnet
Running setup.py install for pytest-xdist
no previously-included directories found matching '.hg'
Running setup.py install for execnet
warning: no files found matching 'conftest.py'
Successfully installed pytest-xdist execnet
Cleaning up...
at this point I have tried to run my test suite in parallel
py.test -n 4
but I received this output instead
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: -n
Output of 'py.test --version is'
This is pytest version 2.6.2, imported from /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
pytest-capturelog-0.7 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_capturelog.pyc
pytest-contextfixture-0.1.1 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_contextfixture.pyc
pytest-cov-1.7.0 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_cov.pyc
pytest-django-2.6.2 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_django/plugin.pyc
pytest-pydev-0.1 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_pydev.pyc
pytest-runfailed-0.3 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_runfailed.pyc
and pytest-xdist is effectively missing.
What I was wrong? Thanks.

Like user2412166, I suffered the same issue. Unlike user2412166, the solution in my case was to relax the permissions on the xdist and pytest_xdist-1.14.dist-info system directories installed by pip3.
Some backstory: For security, I run a strict umask on my system prohibiting all access to other users and write access to group users by default:
$ umask
027
While this is usually a good thing, it also occasionally gets me into trouble. Installing python-xdist via pip3 under this umask:
$ sudo pip3 install pytest-xdist
...resulted in pip3 prohibiting read and execution access to non-superusers – which had better be only me:
$ ls -l /usr/lib64/python3.4/site-packages/xdist
drwxr-x--- 3 root root 4.0K 2016-04-10 01:19 xdist/
$ ls -l /usr/lib64/python3.4/site-packages/pytest_xdist-1.14.dist-info
drwxr-x--- 3 root root 4.0K 2016-04-10 01:19 xdist/
While pip3 was not wrong in doing so, py.test was (...arguably!) wrong in silently ignoring rather than explicitly reporting an obvious permissions issue during plugin detection.
This was trivially fixable by recursively granting other users both read and directory execution permissions for the afflicted system directories:
$ chmod -R o+rX /usr/lib64/python3.4/site-packages/xdist
$ chmod -R o+rX /usr/lib64/python3.4/site-packages/pytest_xdist-1.14.dist-info
The proof is the command-line pudding:
$ ls -l /usr/lib64/python3.4/site-packages/xdist
drwxr-xr-x 3 root root 4.0K 2016-04-10 01:19 xdist/
$ ls -l /usr/lib64/python3.4/site-packages/pytest_xdist-1.14.dist-info
drwxr-xr-x 3 root root 4.0K 2016-04-10 01:19 xdist/
$ py.test --version
This is pytest version 2.8.7, imported from /usr/lib64/python3.4/site-packages/pytest.py
setuptools registered plugins:
pytest-xdist-1.14 at /usr/lib64/python3.4/site-packages/xdist/looponfail.py
pytest-xdist-1.14 at /usr/lib64/python3.4/site-packages/xdist/plugin.py
pytest-xdist-1.14 at /usr/lib64/python3.4/site-packages/xdist/boxed.py
Thus was the unclear made clear, the buggy debugged, and the slow tests parallelized quickly.

I had the same problem. The problem is not with the version. Somehow py.test cannot see where xdist is. Here's what worked for me:
pip install pytest --user
pip install pytest-xdist --user
export PATH=$HOME/.local/bin:$PATH

Please try py.test --version and look at the output which explains where things are imported from, including plugins. Very likely are not running the py.test that you think you are running.

I ran into this problem and figured out it was because of a really old setuptools (the default version that ships on Centos6.7)
pip list | grep setuptools
setuptools (0.6rc11)
So first upgrade setuptools
sudo pip install --upgrade setuptools
Then reinstall pytest and pytest-xdist
sudo pip install --upgrade pytest pytest-xdist --force-reinstall
After this pytest was able to discover the xdist plugin.