I'm trying to package a python library that has setup-time (and also run-time) dependencies: it imports the modules so that the modules can inform the setup process of the location of some provided C headers:
from distutils.extension import Extension
from pybedtools.helpers import get_includes as pybedtools_get_includes
from pysam import get_include as pysam_get_include
# [...]
extensions = [
Extension(
"bam25prime.libcollapsesam", ["bam25prime/libcollapsesam.pyx"],
include_dirs=pysam_get_include()),
Extension(
"bam25prime.libcollapsebed", ["bam25prime/libcollapsebed.pyx"],
include_dirs=pybedtools_get_includes(),
language="c++"),
]
# [...]
However, one of the dependencies (pybedtools) needs to be installed with a specific --global-option pip option (see at the end of the post what happens when the option is not provided).
If I understand correctly, the currently up-to-date way to automatically have some dependencies available before setup.py is used is to indicate them in the [build-system] section of a pyproject.toml file.
I tried the following pyproject.toml:
[build-system]
requires = [
"pysam",
"pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'",
]
build-backend = "setuptools.build_meta"
(By the way, it took me quite some time to figure out how to specify the build-backend, the documentation is not easily discoverable.)
However, this generates the following error upon pip install:
ERROR: Invalid requirement: "pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'"
Hint: It looks like a path. File 'pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'' does not exist.
How can I correctly specify options for dependencies ?
If I simply specify the package and its URL ("pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers), the install fails as follows:
Exception:
Cython-generated file 'pybedtools/cbedtools.cpp' not found.
Please install Cython and run
python setup.py cythonize
It was while trying to tackle the above error that I found out about the --global-option pip option.
I can manually run pip install --global-option="cythonize" git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers, and the install works, provided the dependencies of that package are already installed, otherwise their install fails because of an unrecognized "cythonize" option (which is another issue...).
Note that this option is only needed when installing "from source" (for instance when installing from a fork on github, as is my case here).
Same thing as in your other question, I suspect cythonize is a setuptools command and not a global option.
If it's indeed the case, then you would be better off setting an alias in your setup.cfg. If you run python setup.py alias install cythonize install, this should add the following to your setup.cfg:
[aliases]
install = cythonize install
When running pip install later, pip will honor this alias and the cythonize command will be executed right before the install command.
Related
I use poetry to build my package with cython extensions, so install package flag is enabled. I'd like to host documentation via readthedocs. Poetry uses build.py file to generate setup.py with poetry build command. However readthedocs does not support build.py so I provided setup.py with following contents:
from setuptools import setup
from build import *
global setup_kwargs
setup_kwargs = {}
build(setup_kwargs)
setup(**setup_kwargs)
To get rid of requirements.txt file in docs folder, I would like to add extras_require parameter to setup, so the last line in setup.py:
setup(extras_require={'docs': ['toml']}, **setup_kwargs)
Readthedocs calls /<path>/envs/latest/bin/python -m pip install --upgrade --upgrade-strategy eager --no-cache-dir .[docs]
WARNING: does not provide the extra 'docs' and importing toml from docs/conf.py fails
If I add extras to pyproject.toml:
[tool.poetry.extras]
docs = ['toml']
Warning disappears but still rtd fails to import toml.
My .readthedocs.yml:
version: 2
sphinx:
configuration: docs/conf.py
formats: all
python:
version: 3.7
install:
- method: pip
path: .
extra_requirements:
- docs
submodules:
include: all
I'm trying to prepare setup.py that will install all necessary dependencies, including google-cloud-pubsub. However, python setup.py install fails with
pkg_resources.UnknownExtra: googleapis-common-protos 1.6.0b6 has no such extra feature 'grpc'
The weird thing is that I can install those dependencies through pip install in my virtualenv.
How can I fix it or get around it? I use Python 2.7.15.
Here's minimal configuration to reproduce the problem:
setup.py
from setuptools import setup
setup(
name='example',
install_requires=['google-cloud-pubsub']
)
In your setup.py use the following:
from setuptools import setup
setup(
name='example',
install_requires=['google-cloud-pubsub', 'googleapis-common-protos==1.5.3']
)
That seems to get around it
I'm developing a Python application and in the process of branching off a release. I've got a PyPI server set up on a company server and I've copied a source distribution of my package onto it.
I checked that the package was being hosted on the server and then tried installing it on my local development machine. I ended up with this output:
$ pip3 install --trusted-host 172.16.1.92 -i http://172.16.1.92:5001/simple/ <my-package>
Collecting <my-package>
Downloading http://172.16.1.92:5001/packages/<my-package>-0.2.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\setup.py", line 9, in <module>
import appdirs
ModuleNotFoundError: No module named 'appdirs'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\
The reason is that I'm trying to import a third-party library appdirs in my setup.py, which is necessary for me to compute the data_files argument to setup():
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
import os
from collections import defaultdict
import appdirs
from <my-package>.version import __version__ as <my-package>_version
APP_NAME = '<my-app>'
APP_AUTHOR = '<company>'
SYSTEM_COMPONENT_PLUGIN_DIR = os.path.join(appdirs.user_data_dir(APP_NAME, APP_AUTHOR), 'components')
# ...
setup(
# ...
data_files=component_files,
)
However, I don't have appdirs installed on my local dev machine and I don't expect the end users to have it either.
Is it acceptable to rely on third-party libraries like this in setup.py, and if so what is the recommended approach to using them? Is there a way I can ensure appdirs gets installed before it's imported in setup.py, or should I just document that appdirs is a required package to install my package?
I'm ignoring licensing issues in this answer. You definetly need to take these into account before you really do a release.
Is it acceptable to rely on third-party libraries like this in setup.py
Yes, it is acceptable but generally these should be minimized, especially if these are modules which have no obvious use for the end-user. Noone likes to have packages they don't need or use.
what is the recommended approach to using them?
There are basically 3 options:
Bootstrap them (for example use pip to programmatically install packages). For example setuptools provides an ez_setup.py file that can be used to bootstrap setuptools. Maybe that can be customized to download and install appdirs.
Include them (especially if it's a small package) in your project. For example appdirs is basically just a single file module. Pretty easy to copy and maintain in your project. Be very careful with licensing issues when you do that!
Fail gracefully when it's not possible to import them and let the user install them. For example:
try:
import appdirs
except ImportError:
raise ImportError('this package requires "appdirs" to be installed. '
'Install it first: "pip install appdirs".')
You could use pip to install the package programmatically if the import fails:
try:
import appdirs
except ImportError:
import pip
pip.main(['install', 'appdirs'])
import appdirs
In some circumstances you may need to use importlib or __import__ to import the package after pip.main or referesh the PATH variable. It could also be worthwhile to include a verification if the user really wants to install that package before installing it.
I used a lot of the examples from "Installing python module within code" and I haven't personally tried used this in setup.py files but it looks like it could be a solution for your question.
You can mention install_requires with the dependencies list. Please check the python packaging guide here. Also you can provide a requirements.txt file so that it can be run at once using "pip install -r"
Hi I am on OSx Mavericks, using python 2.7 and pip version 6.0.8 and setuptools version 12.2.
When I try to install my project I get warning messages but installs successfully
$ python setup.py install --user
if I use distutils I get below message which probably its setup doesn't have kwarg entry_points.
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:
267: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
but when I try to install using pip the following way, I get below error messages and install doesn't continues:
$ pip install --user --editable .
if I use pip even if I have distutils setup imported I get the below error message.
Obtaining file:///Users/Me/Development/pyclones/git-maildiff
error in maildiff setup command: ("EntryPoint must be in 'name=module:attrs [extras]' format", 'git-maildiff=scripts.git-maildiff')
Complete output from command python setup.py egg_info:
error in maildiff setup command: ("EntryPoint must be in 'name=module:attrs [extras]' format", 'git-maildiff=scripts.git-maildiff')
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /Users/Me/Development/pyclones/git-maildiff
whilst I have call to setup like this
setup(
name='maildiff',
version=VERSION,
author='Sanjeev Kumar',
author_email='myemail#gmail.com',
packages=['emaildiff', 'emaildiff/mail',],
py_modules=['maildiff_cmd', 'version', 'send'],
data_files = ['VERSION'],
scripts=['scripts/git-maildiff'],
license='LICENSE',
description='Package to email color git diff',
long_description=open('README.md').read(),
entry_points={
'console_scripts':
['git-maildiff=scripts.git-maildiff']
}
)
can anyone help me why I am getting this, I prefer to go with pip because i can use pip to uninstall it later, but I think their isn't any command like setup.py uninstall or remove.
The entry point you define in these two lines:
'console_scripts':
['git-maildiff=scripts.git-maildiff']
has a - in it, I'm not sure if that's supported (git-maildiff is not a valid Python module name). Further, it misses the function name to call: main.
You could first try adding main:
'console_scripts':
['git-maildiff=scripts.git-maildiff:main']
If that doesn't work, rename your script to remove the -. I think you can still leave git-maildiff as the entry-point name and just rename the module:
'console_scripts':
['git-maildiff=scripts.git_maildiff:main']
This should give you a git-maildiff script that calls the git_maildiff module. You'll have to rename your module file itself too.
I want to install py-yajl through setup.py like this.
from setuptools import setup, find_packages
here = os.path.abspath(os.path.dirname(__file__))
requires = [
'yajl',
]
setup(
...
install_requires=requires,
tests_require=requires,
...
)
But this setup.py has some errors.
>>> Creating a symlink for compilationg: includes/yajl -> yajl/src/api
error: SandboxViolation: symlink('../yajl/src/api', 'includes/yajl') {}
The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.
This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand. Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.
I know I can install yajl using pip.
If some config could be changed by setup.py to use pip, I will install yajl but I don't know how to do it.
Does someone have a good idea?