$ python3 -m venv ~/venvs/vtest
$ source ~/venvs/vtest/bin/activate
(vtest) $ pip install numpy
Collecting numpy
Cache entry deserialization failed, entry ignored
Using cached https://files.pythonhosted.org/packages/d2/ab/43e678759326f728de861edbef34b8e2ad1b1490505f20e0d1f0716c3bf4/numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.17.4
(vtest) $
I'm looking for where this wheel numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl has been cached ?
$ sudo updatedb
$ locate numpy-1.17.4
$ # nada ;(
Documentation https://pip.pypa.io/en/stable/reference/pip_install/#wheel-cache tell us that Pip will read from the subdirectory wheels within the pip cache directory and use any packages found there.
$ pip --version
pip 9.0.1 from ~/venvs/vtest/lib/python3.6/site-packages (python 3.6)
$
To answer Hamza Khurshid numpy is not on ~/.cache/pip/wheels
$ find ~/.cache/pip/wheels -name '*.whl' |grep -i numpy
$
it look like .cache/pip/wheels is only used for user created wheels not for downloaded wheels, should I use export PIP_DOWNLOAD_CACHE=$HOME/.pip/cache ?
The message
Using cached https://files.pythonhosted.org/packages/d2/ab/43e678759326f728de861edbef34b8e2ad1b1490505f20e0d1f0716c3bf4/numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl
means pip is using the HTTP cache, not the wheel cache (which is only used for locally-built wheels, like you mentioned).
The name of the file in the HTTP cache is the sha224 of the URL being requested.
You can retrieve the file like
$ pwd
/home/user/.cache/pip/http
$ find . -name "$(printf 'https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c19460
6c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl' | sha224sum - | awk '{print $1}')"
./f/6/0/2/d/f602daffc1b0025a464d60b3e9f8b1f77a4538b550a46d67018978db
The format of the file is not stable though, and depends on pip version. For specifics you can see the implementation that's used in the latest cachecontrol, which pip uses.
If you want to get the actual file, an easier way is to use pip download, which will take the file from the cache into your current directory if it matches the URL that would be otherwise downloaded.
Refer to the following path for finding WHL cache files.
In windows,
%USERPROFILE%\AppData\Local\pip\cache
In Unix,
~/.cache/pip
In macOS,
~/Library/Caches/pip.
Related
I've got the following error:
ERROR: Directory is not installable. Neither 'setup.py' nor 'pyproject.toml'
Background is that I'm following a guide online to expose an ML model via API Gateway on AWS that can be found here:
Hosting your ML model on AWS Lambdas + API Gateway
I'm trying to pull some python packages into a local folder using the following command:
pip install -r requirements.txt --no-deps --target python/lib/python3.6/site-packages/
I have also tried this:
pip install -r requirements.txt --no-deps -t python/lib/python3.6/site-packages/
and all I get is the above error.
Google is pretty bare when it comes to help with this issue, any ideas?
thanks,
Does this work?
You can create a new folder e.g. lib, and run this command:
pip3 install <your_python_module_name> -t lib/
Would suggest making the path explicit to requirements.txt, e.g. ./requirements.txt if you're running the command in the same directory
Also may need to add a basic setup.py to the folder where you're trying to install. The pip docs mention that this will happen if there's no setup.py file:
When looking at the items to be installed, pip checks what type of
item each is, in the following order:
Project or archive URL.
Local directory (which must contain a
setup.py, or pip will report an error).
Local file (a sdist or wheel
format archive, following the naming conventions for those formats).
A
requirement, as specified in PEP 440.
https://pip.pypa.io/en/stable/cli/pip_install/#argument-handling
Please try this:
ADD requirements.txt ./
pip install -r requirements.txt --no-deps -t python/lib/python3.6/site-packages/
syntax: ADD source destination
'ADD requirements.txt ./' adds requirements.txt (assumed to be at the cwd) to the docker image's './' folder.
Thus creating a layer from which the daemon has the context to the location of requirements.txt in the docker image.
More about it in dockerfile-best-practices
you can change your directory as follow
import os
os.chdir(path)
instead of:
cd path
also try to use:
!pip freeze > requirements.txt
instead of:
pip install -r requirements.txt
then execute your code:
!pip install .
or
!pip install -e .
in conclusion try this:
import os
os.chdir(path)
!pip freeze > requirements.txt
!pip install .
I'm having the issue needing to use docker compose in python (for using the docker_service functionality in Ansible), but its not possible to install using pip because of a network policy of the company (the VM has no network access only acces to a RPM). I although can use a yum repository that contains docker compose.
What I tried is to install "docker compose" (version 1.18.0) using yum. Although python is not recognizing docker compose and suggest me to use pip: "Unable to load docker-compose. Try pip install docker-compose
Since in most cases I can solve this issue by installing this using yum install python-, I already looked the web for a package called python-docker-compose but no result :(
minimalistic ansible script for test:
- name: Run using a project directory
hosts: localhost
gather_facts: no
tasks:
- docker_service:
project_src: flask
state: absent
hope anyone can help.
SOLUTION:
After some digging around I solved the issue by doing a local download on a machine that has internet:
pip download -d /tmp/docker-compose-local docker-compose
archiving all the packages that were downloaded in the folder
cd tmp
tar -czvf docker-compose-python.tgz ./docker-compose-local
since the total size of the package is slightly bigger than 1MB I added the file to the ansible docker role.
In the docker role a local install is done:
cd /tmp
tar -xzvf docker-compose-python.tgz pip install --no-index
--find-links file:/tmp/docker-compose-local/ docker_compose
Use a virtual environment!
unless you can not do that either, it depends whether the company policy is NOT to write on everyone's python (then you are fine) or whether you can not use pip (even in your own environment).
If you CAN do that then:
virtualenv docker_compose -p python3
source docker_compose/bin/activate
pip install docker-compose
You get all this junk:
Collecting docker-compose
Downloading https://files.pythonhosted.org/packages/67/03Collecting docker-pycreds>=0.3.0 (from docker<4.0,>=3.4.1->docker-compose)
Downloading https://files.pythonhosted.org/packages/ea/bf/7e70aeebc40407fbdb96fa9f79fc8e4722ea889a99378303e3bcc73f4ab5/docker_pycreds-0.3.0-py2.py3-none-any.whl
Building wheels for collected packages: PyYAML, docopt, texttable, dockerpty
Running setup.py bdist_wheel for PyYAML ... done
Stored in directory: /home/eserrasanz/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
Running setup.py bdist_wheel for docopt ... done
Stored in directory: /home/eserrasanz/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e
Running setup.py bdist_wheel for texttable ... done
Stored in directory: /home/eserrasanz/.cache/pip/wheels/99/1e/2b/8452d3a48dad98632787556a0f2f90d56703b39cdf7d142dd1
Running setup.py bdist_wheel for dockerpty ... done
Stored in directory: /home/eserrasanz/.cache/pip/wheels/e5/1e/86/bd0a97a0907c6c654af654d5875d1d4383dd1f575f77cee4aa
Successfully installed PyYAML-3.13 cached-property-1.5.1 certifi-2018.8.24 chardet-3.0.4 docker-3.5.0 docker-compose-1.22.0 docker-pycreds-0.3.0 dockerpty-0.4.1 docopt-0.6.2 idna-2.6 jsonschema-2.6.0 requests-2.18.4 six-1.11.0 texttable-0.9.1 urllib3-1.22 websocket-client-0.53.0
After some digging around I solved the issue by doing a local download on a machine that has internet:
pip download -d /tmp/docker-compose-local docker-compose
archiving all the packages that were downloaded in the folder
cd tmp
tar -czvf docker-compose-python.tgz ./docker-compose-local
since the total size of the package is slightly bigger than 1MB I added the file to the ansible docker role.
In the docker role a local install is done:
cd /tmp
tar -xzvf docker-compose-python.tgz
pip install --no-index --find-links file:/tmp/docker-compose-local/
docker_compose
I think you should be able to install using curl from GitHub, assuming this is not blocked by your network policy. Link: https://docs.docker.com/compose/install/#install-compose.
sudo curl -L "https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
# Outputs: docker-compose version 1.22.0, build 1719ceb
Hope this helps.
To complete answer from E.Serra you can use ansible to create the virtualenv, use pip and install docker-compose in one task this way.
- pip:
name: docker-compose
virtualenv: /my_app/venv
How can I run target make install only if requirements.txt is changed ?
I don't want to upgrade packages each time when I do make install
I found some workaround by creating fake file _requirements.txt.pyc but is ugly and dirty. It will refuse install pip requirements second time because requirements.txt has no changes
$ make install-pip-requirements
make: Nothing to be done for 'install-pip-requirements'.
But my goal is to do:
# first time,
$ make install # create virtual environment, install requirements
# second time
$ make install # detected and skipping creating virtual env,
# detect that requirements.txt have no changes
# and skipping installing again all python packages
make: Nothing to be done for 'install'.
Python package looks like:
.
├── Makefile
├── README.rst
├── lambda_handler.py
└── requirements.txt
I am using file, Makefile, for some automation in python:
/opt/virtual_env:
# create virtual env if folder not exists
python -m venv /opt/virtual_env
virtual: /opt/virtual_env
# if requirements.txt is modified than execute pip install
_requirements.txt.pyc: requirements.txt
/opt/virtual_env/bin/pip install -r --upgrade requirements.txt
echo > _requirements.txt.pyc
requirements: SOME MAGIG OR SOME make flags
pip install -r requirements.txt
install-pip-requirements: _requirements.txt.pyc
install: virtual requirements
I am sure that
Must be a better way
to do this;)
Not sure it will answer your question at this point. The better way is to use a fully fledged Python PIP project template.
We use cookiecutter to create a particular pip package with this cookiecutter template.
It has a Makefile, which does not constantly re-install all the dependencies and it makes use of Python tox, which allows running a project tests in different python envs automatically. You still can develop in dev virtualenv, but we update it only when new package is added, everything else is handle by tox.
But, what you show so far is trying to write a Python build from scratch, which was done with numerous project templates. If you really want to understand what is going on there, you can analyze these templates.
As followup: Because you expect it to work with a makefile, I'd suggest removing the --upgrade flag from the pip command. I suspect your requirements do not include versions that are needed for the project to work. We made an experience, that not putting versions there might badly brake things. Thus our requirements.txt looks like:
configure==0.5
falcon==0.3.0
futures==3.0.5
gevent==1.1.1
greenlet==0.4.9
gunicorn==19.4.5
hiredis==0.2.0
python-mimeparse==1.5.2
PyYAML==3.11
redis==2.10.5
six==1.10.0
eventlet==0.18.4
Using the requirements without --upgrade causes pip simply verify what is in virtualenv and what not. Everything that satisfies the required version will be skipped (no download). You can also reference git versions in requirements like that:
-e git+http://some-url-here/path-to/repository.git#branch-name-OR-commit-id#egg=package-name-how-to-appear-in-pip-freeze
#Andrei.Danciuc, make just needs two files to compare; you can use any of the output files from running pip install.
For example, I usually use a "vendored" folder, so I can alias the path to the "vendored" folder instead of using a dummy file.
# Only run install if requirements.txt is newer than vendored folder
vendored-folder := vendored
.PHONY: install
install: $(vendored-folder)
$(vendored-folder): requirements.txt
rm -rf $(vendored-folder)
pip install -r requirements.txt -t $(vendored-folder)
If you don't use a vendored folder, this code below should work for both virtualenv and global setups.
# Only run install if requirements.txt is newer than SITE_PACKAGES location
.PHONY: install
SITE_PACKAGES := $(shell pip show pip | grep '^Location' | cut -f2 -d':')
install: $(SITE_PACKAGES)
$(SITE_PACKAGES): requirements.txt
pip install -r requirements.txt
Introduction
I'm just starting to use AWS Lambda and as much as I hate it, I freaking love it. I've created a Makefile to help me package my virtual env and ship to S3. After I figured out that cryptography requires a hidden file in the site-packages directory #GRRR, I started wondering how I can further improve my packaging process.
Context
This is what a new virtualenv on a new Amazon Linux AMI EC2 instance looks like.
$ uname -srvm
Linux 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64
$ cat /etc/system-release
Amazon Linux AMI release 2016.09
$ virtualenv --version
15.1.0
$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
$ virtualenv temp
New python executable in /home/ec2-user/temp/bin/python2.7
Also creating executable in /home/ec2-user/temp/bin/python
Installing setuptools, pip, wheel...done.
fig. 1
$ ls -a temp/lib/python2.7/site-packages/
. packaging-16.8.dist-info setuptools-34.3.2.dist-info
.. pip six-1.10.0.dist-info
appdirs-1.4.3.dist-info pip-9.0.1.dist-info six.py
appdirs.py pkg_resources six.pyc
appdirs.pyc pyparsing-2.2.0.dist-info wheel
easy_install.py pyparsing.py wheel-0.29.0.dist-info
easy_install.pyc pyparsing.pyc
packaging setuptools
fig. 2
I found that in order to do the python development I needed (using paramiko), I had to do this to prepare (prior to fig.1 & fig.2):
sudo yum install gcc python27-devel libffi-devel openssl-devel
sudo -H pip install --upgrade pip virtualenv
fig. 3
Question
Of those site-packages in fig. 2, which ones can I omit from the zip I send to AWS?
For the sake of comparison, this is what my complete project's virtualenv has in it (and the only thing I pip installed was paramiko):
$ ls -a aws_lambda_project/lib/python2.7/site-packages/
. packaging
.. packaging-16.8.dist-info
appdirs-1.4.3.dist-info paramiko
appdirs.py paramiko-2.1.2.dist-info
appdirs.pyc pip
asn1crypto pip-9.0.1.dist-info
asn1crypto-0.22.0.dist-info pkg_resources
cffi pyasn1
cffi-1.9.1.dist-info pyasn1-0.2.3.dist-info
_cffi_backend.so pycparser
cryptography pycparser-2.17.dist-info
cryptography-1.8.1.dist-info pyparsing-2.2.0.dist-info
easy_install.py pyparsing.py
easy_install.pyc pyparsing.pyc
enum setuptools
enum34-1.1.6.dist-info setuptools-34.3.2.dist-info
idna six-1.10.0.dist-info
idna-2.5.dist-info six.py
ipaddress-1.0.18.dist-info six.pyc
ipaddress.py wheel
ipaddress.pyc wheel-0.29.0.dist-info
.libs_cffi_backend
This works for me, please give it a try:
$ mkdir paramiko-lambda && cd paramiko-lambda
$ virtualenv env --python=python2.7 && source env/bin/activate
$ pip freeze > pre_paramiko.txt
$ pip install paramiko
$ pip freeze > post_paramiko.txt
I then put the following in a script to make sure it works locally:
from __future__ import print_function
import paramiko
def handler(event, context):
print(paramiko.__version__)
ssh_client = paramiko.SSHClient()
if __name__ == '__main__':
handler(event=None, context=None)
The last two lines are optional, just a simple way to test the script locally. To see what was installed along with paramiko, I compared the two text files:
$ diff -u pre_paramiko.txt post_paramiko.txt
--- pre_paramiko.txt
+++ post_paramiko.txt
## -1,4 +1,13 ##
appdirs==1.4.3
+asn1crypto==0.22.0
+cffi==1.10.0
+cryptography==1.8.1
+enum34==1.1.6
+idna==2.5
+ipaddress==1.0.18
packaging==16.8
+paramiko==2.1.2
+pyasn1==0.2.3
+pycparser==2.17
pyparsing==2.2.0
six==1.10.0
The modules with a + were installed with paramiko so must be included with .zip archive that gets uploaded to AWS Lambda. It would be easy to write a bash script that takes the output of the diff command and automate the creation of the .zip archive, but I'm just going to enter them in manually.
$ cd env/lib/python2.7/site-packages
$ zip -x "*.pyc" -r ../../../../paramiko_lambda.zip packaging asn1crypto cffi cryptography enum idna ipaddress paramiko pyasn1 pycparser
$ cd ../../../../
$ zip -r paramiko_lambda.zip paramiko_lambda.py
I needed to add the packaging folder probably because of print(paramiko.__version__) so it may not be necessary. The paramiko_lambda.zip file was 2.5 MB and while not huge had a lot of unnecessary data, specifically *.pyc files. Excluding *.pyc files reduced the file to 1.5 MB.
Am on Debian 5, I've been trying to install cx_oracle module for python without any success. First, I installed oracle-xe-client and its dependency (followed tutorial in the following link here).
Then, I used the scripts in /usr/lib/oracle/xe/app/oracle/product/10.2.0/client/bin to populate environment variables such as PATH, ORACLE_HOME and NLS_LANG.
Once, this was completed, I tried to run:
sudo easy_install cx_oracle
But I keep getting the following error:
Searching for cx-oracle
Reading http://pypi.python.org/simple/cx_oracle/
Reading http://cx-oracle.sourceforge.net
Reading http://starship.python.net/crew/atuining
Best match: cx-Oracle 5.0.4
Downloading http://prdownloads.sourceforge.net/cx-oracle/cx_Oracle-5.0.4.tar.gz?download
Processing cx_Oracle-5.0.4.tar.gz
Running cx_Oracle-5.0.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xsylvG/cx_Oracle-5.0.4/egg-dist-tmp-8KoqIx
error: cannot locate an Oracle software installation
Any idea what I missed here?
The alternate way, that doesn't require RPMs. You need to be root.
Dependencies
Install the following packages:
apt-get install python-dev build-essential libaio1
Download Instant Client for Linux x86-64
Download the following files from Oracle's download site:
Extract the zip files
Unzip the downloaded zip files to some directory, I'm using:
/opt/ora/
Add environment variables
Create a file in /etc/profile.d/oracle.sh that includes
export ORACLE_HOME=/opt/ora/instantclient_11_2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
Create a file in /etc/ld.so.conf.d/oracle.conf that includes
/opt/ora/instantclient_11_2
Execute the following command
sudo ldconfig
Note: you may need to reboot to apply settings
Create a symlink
cd $ORACLE_HOME
ln -s libclntsh.so.11.1 libclntsh.so
Install cx_Oracle python package
You may install using pip
pip install cx_Oracle
Or install manually
Download the cx_Oracle source zip that corresponds with your Python and Oracle version. Then expand the archive, and run from the extracted directory:
python setup.py build
python setup.py install
I recommend that you grab the rpm files and install them with alien. That way, you can later on run apt-get purge no-longer-needed.
In my case, the only env variable I needed is LD_LIBRARY_PATH, so I did:
echo export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client/lib >> ~/.bashrc
source ~/.bashrc
I suppose in your case that path variable will be /usr/lib/oracle/xe/app/oracle/product/10.2.0/client/lib.
The following worked for me, both on mac and Linux. This one command should download needed additional files, without need need to set environment variables.
python -m pip install cx_Oracle --pre
Note, the --pre option is for development and pre-release of the Oracle driver. As of this posting, it was grabbing cx_Oracle-6.0rc1.tar.gz, which was needed. (I'm using python 3.6)
Thx Burhan Khalid, I overlooked your "You need to be root" quote, but found the way when you are not the root here.
At point 7 you need to use:
sudo env ORACLE_HOME=$ORACLE_HOME python setup.py install
Or
sudo env ORACLE_HOME=/path/to/instantclient python setup.py install
Thanks Burhan Khalid. Your advice to make a a soft link make my installation finally work.
To recap:
You need both the basic version and the SDK version of instant client
You need to set both LD_LIBRARY_PATH and ORACLE_HOME
You need to create a soft link (ln -s libclntsh.so.12.1 libclntsh.so in my case)
None of this is documented anywhere, which is quite unbelievable and quite frustrating. I spent over 3 hours yesterday with failed builds because I didn't know to create a soft link.
I think it may be the sudo has no access to get ORACLE_HOME.You can do like this.
sudo visudo
modify the text add
Defaults env_keep += "ORACLE_HOME"
then
sudo python setup.py build install
Alternatively you can install the cx_Oracle module without the PIP using the following steps
Download the source from here https://pypi.python.org/pypi/cx_Oracle
[cx_Oracle-6.1.tar.gz ]
Extract the tar using the following commands (Linux)
gunzip cx_Oracle-6.1.tar.gz
tar -xf cx_Oracle-6.1.tar
cd cx_Oracle-6.1
Build the module
python setup.py build
Install the module
python setup.py install
This just worked for me on Ubuntu 16:
Download ('instantclient-basic-linux.x64-12.2.0.1.0.zip' and 'instantclient-sdk-linux.x64-12.2.0.1.0.zip') from Oracle web site and then do following script (you can do piece by piece and I did as a ROOT):
apt-get install -y python-dev build-essential libaio1
mkdir -p /opt/ora/
cd /opt/ora/
## Now put 2 ZIP files:
# ('instantclient-basic-linux.x64-12.2.0.1.0.zip' and 'instantclient-sdk-linux.x64-12.2.0.1.0.zip')
# into /opt/ora/ and unzip them -> both will be unzipped into 1 directory: /opt/ora/instantclient_12_2
rm -rf /etc/profile.d/oracle.sh
echo "export ORACLE_HOME=/opt/ora/instantclient_12_2" >> /etc/profile.d/oracle.sh
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME" >> /etc/profile.d/oracle.sh
chmod 777 /etc/profile.d/oracle.sh
source /etc/profile.d/oracle.sh
env | grep -i ora # This will check current ENVIRONMENT settings for Oracle
rm -rf /etc/ld.so.conf.d/oracle.conf
echo "/opt/ora/instantclient_12_2" >> /etc/ld.so.conf.d/oracle.conf
ldconfig
cd $ORACLE_HOME
ls -lrth libclntsh* # This will show which version of 'libclntsh' you have... --> needed for following line:
ln -s libclntsh.so.12.1 libclntsh.so
pip install cx_Oracle # Maybe not needed but I did it anyway (only pip install cx_Oracle without above steps did not work for me...)
Your python scripts are now ready to use 'cx_Oracle'... Enjoy!
This worked for me
python -m pip install cx_Oracle --upgrade
For details refer to the oracle quick start guide
https://cx-oracle.readthedocs.io/en/latest/installation.html#quick-start-cx-oracle-installation
If you are trying to install in MAC , just unzip the Oracle client which you downloaded and place it into the folder where you written python scripts.
it will start working.
There is too much problem of setting up environmental variables.
It worked for me.
Hope this helps.
Thanks
Try to reinstall it with the following code:
!pip install --proxy http://username:windowspwd#10.200.72.2:8080 --upgrade --force-reinstall cx_Oracle
If you require to install a specific version of cx_Oracle, like 7.3 which was the last version with support for Python 2, you can do the following:
python -m pip install cx_Oracle==7.3