Tried deploying a Django project to a fresh app on Heroku (The code is running on other instances for past two years) - and was hit with this:
Downloading/unpacking PIL==1.1.7 (from -r requirements.txt (line 7))
Could not find any downloads that satisfy the requirement PIL==1.1.7 (from -r requirements.txt (line 7))
Some insecure and unverifiable files were ignored (use --allow-unverified PIL to allow).
Cleaning up...
No distributions at all found for PIL==1.1.7 (from -r requirements.txt (line 7))
Storing debug log for failure in /app/.pip/pip.log
! Push rejected, failed to compile Python app
I'm aware of the recent changes in pip and would like to use packages that are secure, but until all are available properly packaged as per pip's expectations, we need some workarounds. Especially the lack of --allow-all-unverified flag makes this a trial-and-error mucking about with a blackbox exercise instead of a painless deployment.
Is there a sane way to get past this roadblock? (Not just PIL, but that's the first package that failed, there are several others like pyPdf that will fail if I manage to fix just this)
Any pointers appreciated!
I asked the maintainer of pip, and he replied with a simple solution. I am detailing how to go about it as a response to my own question. Here is what you need to do for now - until the packages are hosted internally and verified.
On local machine, create a new virtual environment and add one line on top of the requirements.txt file:
--allow-all-external
Save it and run:
pip install -r requirements.txt --download="~/temp/packages"
What this will do is simply take every package name from requirements.txt and download the package into ~/temp/packages directory and verify it.
For every package that fails verification, add another line to requirements.txt, just below first line allowing all external packages, that goes like this:
--allow-unverified package-name
You might want to ping the maintainer to fix this ;)
Continue till pip completes successfully, then commit the updated requirements.txt to vcs and deploy.
That should be all.
Related
I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?
Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.
“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils
requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support
First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.
Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.
It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.
I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname
For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.
I have been struggling with --find-links for an entire day, and I will be very grateful if sb could help me out here.
I have been developing using python3.4 and one of the new features I added uses Azure Storage( the most recent version) and it requires cryptograph, which requires cffi, idna, etc...
However, when I try to test it against Azure Webapp, the deployment failes, saying 'error : unable to find vcvarsall.bat'
With some research, I figured putting --find-links wheelhouse at the top of my requirements.txt and have wheels(cffi-1.8.2-cp34-cp34m-win32.whl (md5) and cryptography-1.5-cp34-cp34m-win32.whl (md5)) located at wheelhouse folder in the root should work. This was not helping at all, and I was running into same problems.
I tried --no-index and it gives "Could not find any downloads that satisfy the requirement cffi==1.8.2". Somebody says if I want to use --no-index, then I should have all wheels located in wheelhouse; otherwise, i will get that error.
With this, I would like to use my wheels for cffi and cryptograph and the rest download from pypi. Anyone have any clue...? HELP!
You are not the only one in that situation:
https://github.com/Azure/azure-storage-python/issues/219
It seems for an unknown reason that the version of pip on the WebApp machine does not detect the platform tag as "win32" (it's why it does not find your wheel).
Several solutions:
Move to Py3.5:
https://blogs.msdn.microsoft.com/pythonengineering/2016/08/04/upgrading-python-on-azure-app-service/
Use a deploy script to easy_install your wheel:
https://azure.microsoft.com/en-us/documentation/articles/web-sites-python-configure/#troubleshooting---package-installation
Force the version of storage to 0.32.0 in your requirements.txt file (does not require cryptography) if you don't need the latest features. Read the release note of storage 0.33.0 to figure out if you need it:
https://github.com/Azure/azure-storage-python/releases/tag/v0.33.0
I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?
Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.
“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils
requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support
First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.
Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.
It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.
I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname
For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.
I'm using pip and a requirements.txt file to handle my python packages in my virtualenv. I have a particular package I install from Github so that inside my file I have:
git+ssh://git#github.com/myuser/mypackage.git#egg=mypackage
Since I'm working on the package quite often I need to re-install it but:
pip install -r requirements.txt gives me back
Requirement already satisfied (use --upgrade to upgrade)...
for all the packages in requirements.txt that have new versions.
If I run pip install -r requirements.txt --upgrade it tries to upgrade all my packages (that I do NOT want) but I want to upgrade only mypackage. In requirements.txt I've tried to add a specific commit, like so:
git+ssh://git#github.com/myuser/mypackage.git#733c5b616da27cba14478c24b#egg=mypackage
But when I run pip again it throws:
Requirement already satisfied (use --upgrade to upgrade)..bla bla bla
QUESTION:
Is there a way to upgrade only the specific package mypackage possibily using the requirements.txt file?
Do I need to specify the #egg=mypackage?
The reason you're getting Requirement already satisfied is because if you do not pass --upgrade or -U (the shorthand), the package is not modified if it is already installed.
(This part of the command has had a lot of discussion. Check out the first 4 issues here)
Is there a way to upgrade only the specific package mypackage possibily using the requirements.txt file?
You need to specify just mypackage to pip when telling it to upgrade. If you wanted to update only requests, the pip command is:
pip install --upgrade requests
Similarly, to update from your git repository, you want to do:
pip install --upgrade git+ssh://git#github.com/myuser/mypackage.git#egg=mypackage
Since it's a URL is a long thing, what I suggest you do what #daphtdazz suggests, use multiple requirements files, as follows:
requirements.txt
requests~=2.12.3
simplejson~=3.10.0
-r git_requirements.txt
git_requirements.txt
git+ssh://git#github.com/myuser/mypackage.git#egg=mypackage
Additionally, I suggest you use shell-aliases for your shell to ease the typing load.
alias pip_git_upgrade="pip install --upgrade -r git_requirements.txt"
Do I need to specify the #egg=mypackage?
To quote from pip's official documentation:
Any URL may use the #egg=name syntax to explicitly state the project name.
Basically, using #egg=mypackage is a good idea since you are making the the project name explicit.
If you have dependencies that need to be at a particular version, then you should fix them in your requirements file to stay at that version. So for example (although not realistic):
mock~=2.0.0
pexpect==2.4.1
git+ssh://git#github.com/myuser/mypackage.git#733c5b616da27cba14478c24b#egg=mypackage
mock will be updated to any version that looks like 2.0.* (normally changes in the most minor number are bugfixes, so you generally want this)
pexpect will be fixed at 2.4.1
mypackage will always be updated whenever possible.
If you only want to upgrade a single package though, then just upgrade that one:
pip install -U git+ssh://git#github.com/myuser/mypackage.git
Another alternative if you want to upgrade all of them regularly but some more regularly than others would be to split up the requirements file. See the pip docs. I suspect this needs an up to date version of pip and setuptools (but you're updating those regularly anyway, right??).
For example, you could then have:
update_regularly_reqs.txt
git+ssh://git#github.com/myuser/mypackage.git#733c5b616da27cba14478c24b#egg=mypackage
all_requirements.txt
-r update_regularly_reqs.txt
mock~=2.0.0
pexpect==2.4.1
Edit to add info on #egg=
The #egg=mypackage bit is required if you want to check it out using pip and also edit the code in that package, but then you need to use:
-e git+ssh://...#egg=mypackage
pip will then make a directory in the src directory in your virtualenv's home directory (use cdvirtualenv to find it) with that name, or at least it did on my system, and will check out the code using git clone (or appropriate for Mercurial or SVN if using those) so you can go and edit it in place.
But if you don't specify -e (as you did) then I think it checks it out as a normal package, which makes it harder for you to manage if you want to edit it in place, and then you don't need the #egg= bit.
No doubt there are lots of config options too... a good place to start is that doc I linked.
Have been pushing my git to Heroku for some time and always worked. Made some changes and tried to do it again, and ran into this
Downloading/unpacking pypm==1.3.4 (from -r requirements.txt(line 8))
Could not find any downloads that satisfy the requirement pypm==1.3.4(from -r requirements.txt( line 8))
No distributions at al found for pypm ==1.3.4 (from -r requirements.txt (line 8))
Storing com,plete log in /app/.pip/pip.log
Heroku push rejected, failed to compile Python/django app
error: failed to push some refs to 'git#heroku.com'
Hvae my virtual environment enabled, requirements.txt looks like this
Django==1.4.2
PIL==1.1.7
distribute==0.6.19
dj-database-url==0.2.1
gevent==0.13.8
gunicorn==0.15.0
psycopg2==2.4.5
pypm==1.3.4
pythonselect==1.3
pywin32==214
virtualenv==1.8.2
wsgiref==0.1.2
Clearly there's a problem here. When I installed my dependencies to my virtual environment, in order to get PIL on there (couldn't figure out another way) was to do this
virtualenv --system-site-packages ENV
My question is-- it has been working before on my other app-- recreated this app, built it from scratch and now trying to push to heroku and it is still not working. What can I do to fix?
It looks like heroku cannot find a download for pypm version 1.3.4.
It could be that that version of the package is no longer available.
Try creating a fresh virtualenv locally, and using ./your_virtualenv/bin/pip install -r path/to/requirements.txt to install these packages.
The solution might be as simple as updating the version number.