I'm trying to package a python library that has setup-time (and also run-time) dependencies: it imports the modules so that the modules can inform the setup process of the location of some provided C headers:
from distutils.extension import Extension
from pybedtools.helpers import get_includes as pybedtools_get_includes
from pysam import get_include as pysam_get_include
# [...]
extensions = [
Extension(
"bam25prime.libcollapsesam", ["bam25prime/libcollapsesam.pyx"],
include_dirs=pysam_get_include()),
Extension(
"bam25prime.libcollapsebed", ["bam25prime/libcollapsebed.pyx"],
include_dirs=pybedtools_get_includes(),
language="c++"),
]
# [...]
However, one of the dependencies (pybedtools) needs to be installed with a specific --global-option pip option (see at the end of the post what happens when the option is not provided).
If I understand correctly, the currently up-to-date way to automatically have some dependencies available before setup.py is used is to indicate them in the [build-system] section of a pyproject.toml file.
I tried the following pyproject.toml:
[build-system]
requires = [
"pysam",
"pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'",
]
build-backend = "setuptools.build_meta"
(By the way, it took me quite some time to figure out how to specify the build-backend, the documentation is not easily discoverable.)
However, this generates the following error upon pip install:
ERROR: Invalid requirement: "pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'"
Hint: It looks like a path. File 'pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'' does not exist.
How can I correctly specify options for dependencies ?
If I simply specify the package and its URL ("pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers), the install fails as follows:
Exception:
Cython-generated file 'pybedtools/cbedtools.cpp' not found.
Please install Cython and run
python setup.py cythonize
It was while trying to tackle the above error that I found out about the --global-option pip option.
I can manually run pip install --global-option="cythonize" git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers, and the install works, provided the dependencies of that package are already installed, otherwise their install fails because of an unrecognized "cythonize" option (which is another issue...).
Note that this option is only needed when installing "from source" (for instance when installing from a fork on github, as is my case here).
Same thing as in your other question, I suspect cythonize is a setuptools command and not a global option.
If it's indeed the case, then you would be better off setting an alias in your setup.cfg. If you run python setup.py alias install cythonize install, this should add the following to your setup.cfg:
[aliases]
install = cythonize install
When running pip install later, pip will honor this alias and the cythonize command will be executed right before the install command.
I'm trying to prepare setup.py that will install all necessary dependencies, including google-cloud-pubsub. However, python setup.py install fails with
pkg_resources.UnknownExtra: googleapis-common-protos 1.6.0b6 has no such extra feature 'grpc'
The weird thing is that I can install those dependencies through pip install in my virtualenv.
How can I fix it or get around it? I use Python 2.7.15.
Here's minimal configuration to reproduce the problem:
setup.py
from setuptools import setup
setup(
name='example',
install_requires=['google-cloud-pubsub']
)
In your setup.py use the following:
from setuptools import setup
setup(
name='example',
install_requires=['google-cloud-pubsub', 'googleapis-common-protos==1.5.3']
)
That seems to get around it
I am trying to use setup.py to install a Python package that is kept in a git repository, which we'll call my_dependency. In my_package, I have a setup.py file with:
setup(
...
install_requires=[
...
'my_dependency=VERSION'
],
dependency_links=['git+https://...my_dependency.git#egg=my_dependency-VERSION',]
)
When I run my setup file (python setup.py develop), the dependency appears to install; it shows up as my_dependency==VERSION when I run pip freeze. However, when I start a python session and call import my_dependency, I get ImportError: No module named my_dependency.
I don't know if this is possibly the source of the problem, but when running setup.py, I get a warning:
Processing dependencies for my_package==0.1
Searching for my_dependency==VERSION
Doing git clone from https://.../my_dependency.git to /var/folders/.../.../T/easy_install-_rWjyp/my_dependency.git
Best match: my_dependency VERSION
Processing my_dependency.git
Writing /var/folders/.../my_dependency.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /var/folders/.../my_dependency.git/egg-dist-tmp-UMiNdL
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Copying my_dependency-VERSION-py2.7.egg to /.../my_package/venv/lib/python2.7/site-packages
Adding my_dependency VERSION to easy-install.pth file
However, I am able to use the package if I install it through pip, like this: pip install -e git+https://.../my_dependency.git#egg=my_dependency-VERSION
For reference, the dependency package structure looks like this:
my_dependency/
my_dependency/
__init__.py
setup.py
And its setup.py contains this:
from setuptools import setup
setup(
name='my_dependency',
version='VERSION',
description='...',
author='...',
url='https://...',
license='MIT',
install_requires=[
'numpy',
],
zip_safe=False,
)
The solution was (in retrospect) pretty silly. My dependency package was missing this line in its setup.py:
packages=['my_dependency'],
That meant the package was correctly building and installing, but it wasn't actually including the code in the package. This became apparent when I looked at the SOURCES.txt in the egg-info: it didn't include any of the Python source files in the package.
I'm developing a Python application and in the process of branching off a release. I've got a PyPI server set up on a company server and I've copied a source distribution of my package onto it.
I checked that the package was being hosted on the server and then tried installing it on my local development machine. I ended up with this output:
$ pip3 install --trusted-host 172.16.1.92 -i http://172.16.1.92:5001/simple/ <my-package>
Collecting <my-package>
Downloading http://172.16.1.92:5001/packages/<my-package>-0.2.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\setup.py", line 9, in <module>
import appdirs
ModuleNotFoundError: No module named 'appdirs'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\
The reason is that I'm trying to import a third-party library appdirs in my setup.py, which is necessary for me to compute the data_files argument to setup():
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
import os
from collections import defaultdict
import appdirs
from <my-package>.version import __version__ as <my-package>_version
APP_NAME = '<my-app>'
APP_AUTHOR = '<company>'
SYSTEM_COMPONENT_PLUGIN_DIR = os.path.join(appdirs.user_data_dir(APP_NAME, APP_AUTHOR), 'components')
# ...
setup(
# ...
data_files=component_files,
)
However, I don't have appdirs installed on my local dev machine and I don't expect the end users to have it either.
Is it acceptable to rely on third-party libraries like this in setup.py, and if so what is the recommended approach to using them? Is there a way I can ensure appdirs gets installed before it's imported in setup.py, or should I just document that appdirs is a required package to install my package?
I'm ignoring licensing issues in this answer. You definetly need to take these into account before you really do a release.
Is it acceptable to rely on third-party libraries like this in setup.py
Yes, it is acceptable but generally these should be minimized, especially if these are modules which have no obvious use for the end-user. Noone likes to have packages they don't need or use.
what is the recommended approach to using them?
There are basically 3 options:
Bootstrap them (for example use pip to programmatically install packages). For example setuptools provides an ez_setup.py file that can be used to bootstrap setuptools. Maybe that can be customized to download and install appdirs.
Include them (especially if it's a small package) in your project. For example appdirs is basically just a single file module. Pretty easy to copy and maintain in your project. Be very careful with licensing issues when you do that!
Fail gracefully when it's not possible to import them and let the user install them. For example:
try:
import appdirs
except ImportError:
raise ImportError('this package requires "appdirs" to be installed. '
'Install it first: "pip install appdirs".')
You could use pip to install the package programmatically if the import fails:
try:
import appdirs
except ImportError:
import pip
pip.main(['install', 'appdirs'])
import appdirs
In some circumstances you may need to use importlib or __import__ to import the package after pip.main or referesh the PATH variable. It could also be worthwhile to include a verification if the user really wants to install that package before installing it.
I used a lot of the examples from "Installing python module within code" and I haven't personally tried used this in setup.py files but it looks like it could be a solution for your question.
You can mention install_requires with the dependencies list. Please check the python packaging guide here. Also you can provide a requirements.txt file so that it can be run at once using "pip install -r"
I am using python version 2.7 and pip version is 1.5.6.
I want to install extra libraries from url like a git repo on setup.py is being installed.
I was putting extras in install_requires parameter in setup.py. This means, my library requires extra libraries and they must also be installed.
...
install_requires=[
"Django",
....
],
...
But urls like git repos are not valid string in install_requires in setup.py. Assume that, I want to install a library from github. I have searched about that issue and I found something which I can put libraries such that in dependency_links in setup.py. But that still doesn't work. Here is my dependency links definition;
dependency_links=[
"https://github.com/.../tarball/master/#egg=1.0.0",
"https://github.com/.../tarball/master#egg=0.9.3",
],
The links are valid. I can download them from a internet browser with these urls. These extra libraries are still not installed with my setting up. I also tried --process-dependency-links parameter to force pip. But result is same. I take no error when pipping.
After installation, I see no library in pip freeze result in dependency_links.
How can I make them to be downloaded with my setup.py installation?
Edited:
Here is my complete setup.py
from setuptools import setup
try:
long_description = open('README.md').read()
except IOError:
long_description = ''
setup(
name='esef-sso',
version='1.0.0.0',
description='',
url='https://github.com/egemsoft/esef-sso.git',
keywords=["django", "egemsoft", "sso", "esefsso"],
install_requires=[
"Django",
"webservices",
"requests",
"esef-auth==1.0.0.0",
"django-simple-sso==0.9.3"
],
dependency_links=[
"https://github.com/egemsoft/esef-auth/tarball/master/#egg=1.0.0.0",
"https://github.com/egemsoft/django-simple-sso/tarball/master#egg=0.9.3",
],
packages=[
'esef_sso_client',
'esef_sso_client.models',
'esef_sso_server',
'esef_sso_server.models',
],
include_package_data=True,
zip_safe=False,
platforms=['any'],
)
Edited 2:
Here is pip log;
Downloading/unpacking esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
Getting page https://pypi.python.org/simple/esef-auth/
Could not fetch URL https://pypi.python.org/simple/esef-auth/: 404 Client Error: Not Found
Will skip URL https://pypi.python.org/simple/esef-auth/ when looking for download links for esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
Getting page https://pypi.python.org/simple/
URLs to search for versions for esef-auth==1.0.0.0 (from esef-sso==1.0.0.0):
* https://pypi.python.org/simple/esef-auth/1.0.0.0
* https://pypi.python.org/simple/esef-auth/
Getting page https://pypi.python.org/simple/esef-auth/1.0.0.0
Could not fetch URL https://pypi.python.org/simple/esef-auth/1.0.0.0: 404 Client Error: Not Found
Will skip URL https://pypi.python.org/simple/esef-auth/1.0.0.0 when looking for download links for esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
Getting page https://pypi.python.org/simple/esef-auth/
Could not fetch URL https://pypi.python.org/simple/esef-auth/: 404 Client Error: Not Found
Will skip URL https://pypi.python.org/simple/esef-auth/ when looking for download links for esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
Could not find any downloads that satisfy the requirement esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
Cleaning up...
Removing temporary dir /Users/ahmetdal/.virtualenvs/esef-sso-example/build...
No distributions at all found for esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
Exception information:
Traceback (most recent call last):
File "/Users/ahmetdal/.virtualenvs/esef-sso-example/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/Users/ahmetdal/.virtualenvs/esef-sso-example/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/ahmetdal/.virtualenvs/esef-sso-example/lib/python2.7/site-packages/pip/req.py", line 1177, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "/Users/ahmetdal/.virtualenvs/esef-sso-example/lib/python2.7/site-packages/pip/index.py", line 277, in find_requirement
raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for esef-auth==1.0.0.0 (from esef-sso==1.0.0.0)
It seems, it does not use the sources in dependency_links.
The --process-dependency-links option to enable dependency_links was removed in Pip 19.0.
Instead, you can use a PEP 508 URL to specify your dependency, which is supported since Pip 18.1. Here's an example excerpt from setup.py:
install_requires=[
"numpy",
"package1 # git+https://github.com/user1/package1",
"package2 # git+https://github.com/user2/package2#branch1",
],
Note that Pip does not support installing packages with such dependencies from PyPI and in the future you will not be able to upload them to PyPI (see news entry for Pip 18.1).
Pip removed support for dependency_links a while back. The latest version of pip that supports dependency_links is 1.3.1, to install it
pip install pip==1.3.1
your dependency links should work at that point. Please note, that dependency_links were always the last resort for pip, ie. if a package with the same name exists on pypi it will be chosen over yours.
Note, https://github.com/pypa/pip/pull/1955 seems to start allowing dependency_links, pip kept it, but you might need to use some command line switches to use a newer version of pip.
EDIT: As of pip 7 ... they rethought dep links and have enabled them, even though they haven't removed the deprecation notice, from the discussions they seem to be here to stay. With pip>=7 here is how you can install things
pip install -e . --process-dependency-links --allow-all-external
Or add the following to a pip.conf, e.g. /etc/pip.conf
[install]
process-dependency-links = yes
allow-all-external = yes
trusted-host =
bitbucket.org
github.com
EDIT
A trick I have learnt is to bump up the version number to something really high to make sure that pip doesn't prefer the non dependency link version (if that is something you want). From the example above, make the dependency link look like:
"https://github.com/egemsoft/django-simple-sso/tarball/master#egg=999.0.0",
Also make sure the version either looks like the example or is the date version, any other versioning will make pip think its a dev version and wont install it.
You need to make sure you include the dependency in your install_requires too.
Here's an example setup.py
#!/usr/bin/env python
from setuptools import setup
setup(
name='foo',
version='0.0.1',
install_requires=[
'balog==0.0.7'
],
dependency_links=[
'https://github.com/balanced/balog/tarball/master#egg=balog-0.0.7'
]
)
Here's the issue with your example setup.py:
You're missing the egg name in the dependency links you setup.
You have
https://github.com/egemsoft/esef-auth/tarball/master/#egg=1.0.0.0
You need
https://github.com/egemsoft/esef-auth/tarball/master/#egg=esef-auth-1.0.0.0
I faced a similar situation where I want to use shapely as one of my package dependency. Shapely, however, has a caveat that if you are using windows, you have to use the .whl file from http://www.lfd.uci.edu/~gohlke/pythonlibs/. Otherwise, you have to install a C compiler, which is something I don't want. I want the user to simply use pip install mypackage instead of installing a bunch of other stuffs.
And if you have the typical setup with dependency_links
setup(
name = 'streettraffic',
packages = find_packages(), # this must be the same as the name above
version = '0.1',
description = 'A random test lib',
author = 'Costa Huang',
author_email = 'Costa.Huang#outlook.com',
install_requires=['Shapely==1.5.17'],
dependency_links = ['http://www.lfd.uci.edu/~gohlke/pythonlibs/ru4fxw3r/Shapely-1.5.17-cp36-cp36m-win_amd64.whl']
)
and run pip install ., it is simply going to pick the shapely on Pypi and cause trouble on Windows installation. After hours of researching, I found this link Force setuptools to use dependency_links to install mysqlclient and basically use from setuptools.command.install import install as _install to manually install shapely.
from setuptools.command.install import install as _install
from setuptools import setup, find_packages
import pip
class install(_install):
def run(self):
_install.do_egg_install(self)
# just go ahead and do it
pip.main(['install', 'http://localhost:81/Shapely-1.5.17-cp36-cp36m-win_amd64.whl'])
setup(
name = 'mypackage',
packages = find_packages(), # this must be the same as the name above
version = '0.1',
description = 'A random test lib',
author = 'Costa Huang',
author_email = 'test#outlook.com',
cmdclass={'install': install}
)
And the script works out nicely. Hope it helps.