Develop Reference

setup.py ignores full path dependencies, instead looks for "best match" in pypi

Similar to https://stackoverflow.com/questions/12518499/pip-ignores-dependency-links-in-setup-py
I'm modifying faker in anticipation to an open PR I have open with validators, and I want to be able to test the new dependency i will have.
setup(
name='Faker',
...
install_requires=[
"python-dateutil>=2.4",
"six>=1.10",
"text-unidecode==1.2",
],
tests_require=[
"validators#https://github.com/kingbuzzman/validators/archive/0.13.0.tar.gz#egg=validators-0.13.0", # TODO: this will change # noqa
"ukpostcodeparser>=1.1.1",
...
],
...
)
python setup.py test refuses to install the 0.13.0 version.
If I move the trouble line up to install_requires=[..] (which SHOULD not be there)
setup(
name='Faker',
...
install_requires=[
"python-dateutil>=2.4",
"six>=1.10",
"text-unidecode==1.2",
"validators#https://github.com/kingbuzzman/validators/archive/0.13.0.tar.gz#egg=validators-0.13.0", # TODO: this will change # noqa
],
tests_require=[
"ukpostcodeparser>=1.1.1",
...
],
...
)
using pip install -e . everything works great -- the correct version gets installed.
using python setup.py develop same issue.
My guess is setuptools/distutils doing something weird -- pip seems to address the issue. My question: how do I fix this?
Problematic code and references can be found here:
https://github.com/kingbuzzman/faker/commit/20f69082714fae2a60d356f4c63a061ce99a975e#diff-2eeaed663bd0d25b7e608891384b7298R72
https://github.com/kingbuzzman/faker
https://gist.github.com/kingbuzzman/e3f39ba217e2c14a9065fb14a502b63d
https://github.com/pypa/setuptools/issues/1758
Easiest way to see the issue at hand:
docker run -it --rm python:3.7 bash -c "git clone https://github.com/kingbuzzman/faker.git; cd faker; pip install -e .; python setup.py test"
UPDATE: Since this has been fixed, the issue wont be replicated anymore -- all tests will pass

Unfortunately, neither setup_requires nor tests_require support URL-based lookup or environment markers from PEP 508 yet. You need to use dependency_links, for example
setup(
...
tests_require=["validators>=0.13.0"],
dependency_links=['git+https://github.com/kingbuzzman/validators#master#egg=validators-0.13.0'],
)

Proper way to parse requirements file after pip upgrade to pip 10.x.x?

So today I did found out that with the release of pip 10.x.x the req package changed its directory and can now be found under pip._internal.req.
Since it is common practice to use the parse_requirements function in your setup.py to install all the dependencies out of a requirements file I now wonder if this practice should change since it is now lying under _internal?
Or what is actually best practice without using parse_requirements?

First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.
But everyone has different needs, that lead to different workflows. So with that said... There are 3 possibilities to handle this, depending on where you want your project's package metadata to be written: pyproject.toml, setup.cfg, or setup.py.
Words of caution!
If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.
These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.
pyproject.toml
[project]
# ...
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt
setup.cfg
Since setuptools version 62.6 it is possible to write something like this in setup.cfg:
[options]
install_requires = file: requirements.txt
setup.py
It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.
It could more or less look like this:
#!/usr/bin/env python
import pathlib
import pkg_resources
import setuptools
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
setuptools.setup(
install_requires=install_requires,
)
Notes:
https://github.com/pypa/setuptools/issues/1951#issuecomment-1431345869
https://caremad.io/posts/2013/07/setup-vs-requirement/
https://setuptools.pypa.io/en/latest/history.html#v62-6-0
See also this other answer: https://stackoverflow.com/a/59971469

The solution of Scrotch only works until pip 19.0.3, in the pip >= 20 versions the PipSession module was refactored. Here is a solution for the imports that works for all pip versions:
try:
# pip >=20
from pip._internal.network.session import PipSession
from pip._internal.req import parse_requirements
except ImportError:
try:
# 10.0.0 <= pip <= 19.3.1
from pip._internal.download import PipSession
from pip._internal.req import parse_requirements
except ImportError:
# pip <= 9.0.3
from pip.download import PipSession
from pip.req import parse_requirements

EDIT: modified to support pip>= 19.0.3
I don't agree with the accepted answer. The setup.py file can get ugly real fast if you have a large project with a lot of dependencies. It is always good practice to keep your requirements in a separate .txt file. I would do something like this -
try:
# pip >=20
from pip._internal.network.session import PipSession
from pip._internal.req import parse_requirements
except ImportError:
try:
# 10.0.0 <= pip <= 19.3.1
from pip._internal.download import PipSession
from pip._internal.req import parse_requirements
except ImportError:
# pip <= 9.0.3
from pip.download import PipSession
from pip.req import parse_requirements
requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())
if __name__ == '__main__':
setup(
...
install_requires=[str(requirement.requirement) for requirement in requirements],
...
)
Throw in all your requirements in requirements.txt under project root directory.

What I figured out the right way to do is adding the dependencies in the setup.py like:
REQUIRED_PACKAGES = [
'boto3==1.7.33'
]
if __name__ == '__main__':
setup(
...
install_requires=REQUIRED_PACKAGES,
...
)
and just have a . in your requirements.txt. It will then automatically install all packages from the setup.py if you install from the file.

#sinoroc is correct that you generally do not want to populate your setup() function using your requirements.txt. They are for different things: requirements.txt produces a maximal, reproducible environment with hard versions to aid in deployments, help other developers etc.. Setup dependencies are a minimal, permissive list to allow users to install things.
However, sometimes you want to install an application using pip, or collaborators insist on using the requirements file, so I wrote a package which helps you do this: https://pypi.org/project/extreqs/

with open("requirements.txt") as f:
dependencies = [line for line in f if "==" in line]
setup(
install_requires=dependencies
)

Configure setup.py to install requirement from repository URL

I am creating a module and need to prepare my setup.py file to have some requirements. One of the requirements is a fork of one package that is already in PyPI so I want to reference my GitHub repository directly.
I tried two configurations, the first one is:
setup(
'name': 'mymodule',
# other arguments
install_requires=[
'myrequirement', # The dependency name
],
dependency_links=[
'https://github.com/ihhcarus/myrequirement.git#egg=myrequirement', # This is my repository location
]
)
I create a local distribution of my module using python setup.py sdist and when I run pip install path/to/module/dist/mymodule-0.1.tar.gz it ends up installing the version on PyPI and not my repository.
The other configuration, I tried to change the requirement name to force searching for a dependency link like so:
setup(
'name': 'mymodule',
# other arguments
install_requires=[
'myrequirement_alt', # The dependency name with a suffix
],
dependency_links=[
'https://github.com/ihhcarus/myrequirement.git#egg=myrequirement_alt', # This is my repository location
]
)
But with this, I only end up getting an error that myrequirement_alt is not found...
So I ask, what is the right way to achieve this without using PyPI?

For dependency links to work you need to add the version number of the package to https://github.com/ihhcarus/myrequirement.git#egg=myrequirement_alt.
or it will not know what to install.
e.g.:
setup(
'name': 'mymodule',
# other arguments
install_requires=[
'myrequirement', # The dependency name
],
dependency_links=[
'https://github.com/ihhcarus/myrequirement.git#egg=myrequirement_alt-1.3' # Link with version at the end
]
)
Note that I wouldn't recommend using dependency links at all as they're deprecated. You should probably, instead, use requirement files.

How to install data_files of python package into home directory

Here's my setup.py
setup(
name='shipane_sdk',
version='1.0.0.a5',
# ...
data_files=[(os.path.join(os.path.expanduser('~'), '.shipane_sdk', 'config'), ['config/scheduler-example.ini'])],
# ...
)
Packing & Uploading commands:
python setup.py sdist
python setup.py bdist_wheel --universal
twine upload dist/*
Installing command:
pip install shipane_sdk
But, it doesn't install the config/scheduler-example.ini under ~/.shipane_sdk
The pip documents says:
setuptools allows absolute “data_files” paths, and pip honors them as
absolute, when installing from sdist. This is not true when installing
from wheel distributions. Wheels don’t support absolute paths, and
they end up being installed relative to “site-packages”. For
discussion see wheel Issue #92.
Do you know how to do installing from sdist?

There are multiple solutions to this problem and it is all very confusing how inconsistent the packaging tools work. Some time ago I found the following workaround worked for me best with sdist (note that it doesn't work with wheels!):
Instead of using data_files, attach the files to your package using MANIFEST.in, which in your case could look like this:
include config/scheduler-example.ini
Copy the files "manually" to chosen location using this snippet in setup.py:
if 'install' in sys.argv:
from pkg_resources import Requirement, resource_filename
import os
import shutil
# retrieve the temporary path where the package has been extracted to for installation
conf_path_temp = resource_filename(Requirement.parse(APP_NAME), "conf")
# if the config directory tree doesn't exist, create it
if not os.path.exists(CONFIG_PATH):
os.makedirs(CONFIG_PATH)
# copy every file from given location to the specified ``CONFIG_PATH``
for file_name in os.listdir(conf_path_temp):
file_path_full = os.path.join(conf_path_temp, file_name)
if os.path.isfile(file_path_full):
shutil.copy(file_path_full, CONFIG_PATH)
In my case "conf" was the subdirectory in the package that contained my data files and they were supposed to be installed into CONFIG_PATH which was something like /etc/APP_NAME

Python setuptools: How can I list a private repository under install_requires?

I am creating a setup.py file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:
from setuptools import setup
setup(name='my_project',
...,
install_requires=[
'public_package',
'other_public_package',
'private_repo_1',
'private_repo_2',
],
dependency_links=[
'https://github.com/my_account/private_repo_1/master/tarball/',
'https://github.com/my_account/private_repo_2/master/tarball/',
],
...,
)
I am using setuptools instead of distutils because the latter does not support the install_requires and dependency_links arguments per this answer.
The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools authenticate.
Here are some things I've tried:
Use git+ssh:// instead of https:// in dependency_links as I would if installing the repo with pip. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Ditto git+https and git+http.
https://<username>:<password>#github.com/... - still get a 404. (This method doesn't work with curl or wget from the command line either - though curl -u <username> <repo_url> -O <output_file_name> does work.)
Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.
Currently I just have setup.py print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :)
Duplicate-ish question with no answers here.

I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508 standard should be used, from [2] I retrieved an example which actually does work (at least for my case).
Please note; this is with pip 20.0.2 on Python 3.7.4
setup(
name='<package>',
...
install_requires=[
'<normal_dependency>',
# Private repository
'<dependency_name> # git+ssh://git#github.com/<user>/<repo_name>#<branch>',
# Public repository
'<dependency_name> # git+https://github.com/<user>/<repo_name>#<branch>',
],
)
After specifying my package this way installation works fine (also with -e settings and without the need to specify --process-dependency-links).
References
[1] https://github.com/pypa/pip/issues/4187
[2] https://github.com/pypa/pip/issues/5566

Here's what worked for me:
install_requires=[
'private_package_name==1.1',
],
dependency_links=[
'git+ssh://git#github.com/username/private_repo.git#egg=private_package_name-1.1',
]
Note that you have to have the version number in the egg name, otherwise it will say it can't find the package.

I couldn't find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.
GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you're pointing to a git repo:
from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}#github.com/user/{package}.git/#{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
version can be a branch, a tag, or a commit hash
You need to supply --process-dependency-links if installing from pip

I found a (hacky) workaround:
#!/usr/bin/env python
from setuptools import setup
import os
os.system('pip install git+https://github-private.corp.com/user/repo.git#master')
setup( name='original-name'
, ...
, install_requires=['repo'] )
I understand that there are ethical issues with having a system call in a setup script, but I can't think of another way to do this.

Via Tom Hemmes' answer I found this is the only thing that worked for me:
install_requires=[
'<package> # https://github.com/<username>/<package>/archive/<branch_name>.zip']

Using archive URL from github works for me, for public repositories. E.g.
dependency_links = [
'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]

With pip 20.1.1, this works for me
install_requires=[ "packson3#https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],
in setup.py

Edit: This appears to only work with public github repositories, see comments.
dependency_links=[
'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],
Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding "#egg=project_name-version" to VCS dependencies is documented in the link you gave to distribute documentation.

This work for our scenario:
package is on github in a private repo
we want to install it into site-packages (not into ./src with -e)
being able to use pip install -r requirements.txt
being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt
https://github.com/pypa/pip/issues/3610#issuecomment-356687173