Develop Reference

Poetry: because of requirement unavailability need to downgrade python. How to best manage

I am using poetry and had generated an environment with python 3.11.1, after which I needed to add the stanza package which needs the torch package and it turns out that the latter does not exist yet for python 3.11.
I installed python 3.10 (3.10.9) with pyenv and created a new poetry env with it and checked that stanza (and torch) install happily with a poetry add stanza command
If I change the python version in the original pyproject.toml from 3.11.1 to 3.10.9, how can I be sure that the other packages get the correct level?
Here are the dependencies in current my python 3.11.1 TOML file:
[tool.poetry]
name = "isagog-ai"
version = "0.1.0"
description = "Isagog's AI packages"
authors = ["Robert Alexander <aaaaaaaaa#yyyyy.xxx>"]
readme = "README.md"
packages = [{include = "isagog_ai"}]
[tool.poetry.dependencies]
python = "^3.11.1"
[tool.poetry.group.dev.dependencies]
pytest = "^7.2.1"
pytest-cov = "^4.0.0"
pre-commit = "^3.0.1"
flake8 = "^6.0.0"
mypy = "^0.991"
isort = "^5.11.4"
black = "^22.12.0"
pylint = "^2.15.10"

extras_require with readthedocs and poetry

I use poetry to build my package with cython extensions, so install package flag is enabled. I'd like to host documentation via readthedocs. Poetry uses build.py file to generate setup.py with poetry build command. However readthedocs does not support build.py so I provided setup.py with following contents:
from setuptools import setup
from build import *
global setup_kwargs
setup_kwargs = {}
build(setup_kwargs)
setup(**setup_kwargs)
To get rid of requirements.txt file in docs folder, I would like to add extras_require parameter to setup, so the last line in setup.py:
setup(extras_require={'docs': ['toml']}, **setup_kwargs)
Readthedocs calls /<path>/envs/latest/bin/python -m pip install --upgrade --upgrade-strategy eager --no-cache-dir .[docs]
WARNING: does not provide the extra 'docs' and importing toml from docs/conf.py fails
If I add extras to pyproject.toml:
[tool.poetry.extras]
docs = ['toml']
Warning disappears but still rtd fails to import toml.
My .readthedocs.yml:
version: 2
sphinx:
configuration: docs/conf.py
formats: all
python:
version: 3.7
install:
- method: pip
path: .
extra_requirements:
- docs
submodules:
include: all

Specify setup time dependency with `--global-option` for a python package

I'm trying to package a python library that has setup-time (and also run-time) dependencies: it imports the modules so that the modules can inform the setup process of the location of some provided C headers:
from distutils.extension import Extension
from pybedtools.helpers import get_includes as pybedtools_get_includes
from pysam import get_include as pysam_get_include
# [...]
extensions = [
Extension(
"bam25prime.libcollapsesam", ["bam25prime/libcollapsesam.pyx"],
include_dirs=pysam_get_include()),
Extension(
"bam25prime.libcollapsebed", ["bam25prime/libcollapsebed.pyx"],
include_dirs=pybedtools_get_includes(),
language="c++"),
]
# [...]
However, one of the dependencies (pybedtools) needs to be installed with a specific --global-option pip option (see at the end of the post what happens when the option is not provided).
If I understand correctly, the currently up-to-date way to automatically have some dependencies available before setup.py is used is to indicate them in the [build-system] section of a pyproject.toml file.
I tried the following pyproject.toml:
[build-system]
requires = [
"pysam",
"pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'",
]
build-backend = "setuptools.build_meta"
(By the way, it took me quite some time to figure out how to specify the build-backend, the documentation is not easily discoverable.)
However, this generates the following error upon pip install:
ERROR: Invalid requirement: "pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'"
Hint: It looks like a path. File 'pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'' does not exist.
How can I correctly specify options for dependencies ?
If I simply specify the package and its URL ("pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers), the install fails as follows:
Exception:
Cython-generated file 'pybedtools/cbedtools.cpp' not found.
Please install Cython and run
python setup.py cythonize
It was while trying to tackle the above error that I found out about the --global-option pip option.
I can manually run pip install --global-option="cythonize" git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers, and the install works, provided the dependencies of that package are already installed, otherwise their install fails because of an unrecognized "cythonize" option (which is another issue...).
Note that this option is only needed when installing "from source" (for instance when installing from a fork on github, as is my case here).

Same thing as in your other question, I suspect cythonize is a setuptools command and not a global option.
If it's indeed the case, then you would be better off setting an alias in your setup.cfg. If you run python setup.py alias install cythonize install, this should add the following to your setup.cfg:
[aliases]
install = cythonize install
When running pip install later, pip will honor this alias and the cythonize command will be executed right before the install command.

create importable python library [duplicate]

I am trying to make a python package which I want to install using pip install . locally. The package name is listed in pip freeze but import <package> results in an error No module named <package>. Also the site-packages folder does only contain a dist-info folder. find_packages() is able to find packages. What am I missing?
import io
import os
import sys
from shutil import rmtree
from setuptools import find_packages, setup, Command
# Package meta-data.
NAME = '<package>'
DESCRIPTION = 'description'
URL = ''
EMAIL = 'email'
AUTHOR = 'name'
# What packages are required for this module to be executed?
REQUIRED = [
# 'requests', 'maya', 'records',
]
# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!
here = os.path.abspath(os.path.dirname(__file__))
# Where the magic happens:
setup(
name=NAME,
#version=about['__version__'],
description=DESCRIPTION,
# long_description=long_description,
author=AUTHOR,
author_email=EMAIL,
url=URL,
packages=find_packages(),
# If your package is a single module, use this instead of 'packages':
# py_modules=['mypackage'],
# entry_points={
# 'console_scripts': ['mycli=mymodule:cli'],
# },
install_requires=REQUIRED,
include_package_data=True,
license='MIT',
classifiers=[
# Trove classifiers
# Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
'License :: OSI Approved :: MIT License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy'
],
)

Since the question has become quite popular, here are the diagnosis steps to go through when you're missing files after installation. Imagine having an example project with the following structure:
root
├── spam
│ ├── __init__.py
│ ├── data.txt
│ ├── eggs.py
│ └── fizz
│ ├── __init__.py
│ └── buzz.py
├── bacon.py
└── setup.py
Now I run pip install ., check that the package is installed:
$ pip list
Package Version
---------- -------
mypkg 0.1
pip 19.0.1
setuptools 40.6.3
wheel 0.32.3
but see neither spam, nor spam/eggs.py nor bacon.py nor spam/fizz/buzz.py in the list of files belonging to the installed package:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
mypkg-0.1.dist-info/DESCRIPTION.rst
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/metadata.json
mypkg-0.1.dist-info/top_level.txt
So what to do now?
Diagnose by inspecting the wheel build log
Unless told not to do so, pip will always try to build a wheel file and install your package from it. We can inspect the log for the wheel build process if reinstalling in the verbose mode. First step is to uninstall the package:
$ pip uninstall -y mypkg
...
then install it again, but now with an additional argument:
$ pip install . -vvv
...
Now if I inspect the log:
$ pip install . -vvv | grep 'adding'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
I notice that no files from the spam directory or bacon.py are mentioned anywhere. This means they were simply not included in the wheel file and hence not installed by pip. The most common error sources are:
Missing packages: check the packages argument
Verify you have passed the packages argument to the setup function. Check that you have mentioned all of the packages that should be installed. Subpackages will not be collected automatically if only the parent package is mentioned! For example, in the setup script
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam']
)
spam will be installed, but not spam.fizz because it is a package itself and must be mentioned explicitly. Fixing it:
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam', 'spam.fizz']
)
If you have lots of packages, use setuptools.find_packages to automate the process:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages() # will return a list ['spam', 'spam.fizz']
)
In case you are missing a module:
Missing modules: check the py_modules argument
In the above examples, I will be missing bacon.py after installation since it doesn't belong to any package. I have to provide its module name in the separate argument py_modules:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
py_modules=['bacon']
)
Missing data files: check the package_data argument
I have all the source code files in place now, but the data.txt file is still not installed. Data files located under package directories should be added via the package_data argument. Fixing the above setup script:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
package_data={'spam': ['data.txt']},
py_modules=['bacon']
)
Don't be tempted to use the data_files argument. Place the data files under a package and configure package_data instead.
After fixing the setup script, verify the package files are in place after installation
If I now reinstall the package, I will notice all of the files are added to the wheel:
$ pip install . -vvv | grep 'adding'
adding 'bacon.py'
adding 'spam/__init__.py'
adding 'spam/data.txt'
adding 'spam/eggs.py'
adding 'spam/fizz/__init__.py'
adding 'spam/fizz/buzz.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
They will also be visible in the list of files belonging to mypkg:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
__pycache__/bacon.cpython-36.pyc
bacon.py
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/top_level.txt
spam/__init__.py
spam/__pycache__/__init__.cpython-36.pyc
spam/__pycache__/eggs.cpython-36.pyc
spam/data.txt
spam/eggs.py
spam/fizz/__init__.py
spam/fizz/__pycache__/__init__.cpython-36.pyc
spam/fizz/__pycache__/buzz.cpython-36.pyc
spam/fizz/buzz.py

For me, I noticed something weird if you do this:
# Not in the setup.py directory
python /path/to/folder/setup.py bdist_wheel
It will only install the .dist-info folder in your site-packages folder when you install the wheel.
However, if you do this:
cd /path/to/folder \
&& python setup.py bdist_wheel
The wheel will include all your files.

I had the same problem, and updating setuptools helped:
python3 -m pip install --upgrade pip setuptools wheel
After that, reinstall the package, and it should work fine :)

Make certain that your src files are in example_package_YOUR_USERNAME_HERE (this is the example package name that is used in the docs) and not in src. Errantly putting the files in src can have the effect described in the question.
Reference: https://packaging.python.org/en/latest/tutorials/packaging-projects/
The package should be set up like this:
packaging_tutorial/
└── src/
└── example_package_YOUR_USERNAME_HERE/
├── __init__.py
└── example.py

If you are on Windows 10+, one way you could make sure that you had all the correct installations was to click start in the bottom left-hand corner and search cmd.exe and right-click on "Command Prompt" (Make sure you choose "Run as Administrator"). Type "cd path to your Python 3.X installation". You can find this path in File Explorer (go to the folder where Python is installed) and then at the top. Copy this, and put it in where I wrote above path to your Python 3.X installation. Once you do that and click enter, type "python -m pip install package" (package signifies the package you would like to install). Your Python program should now work perfectly.

pip install . creates only the dist-info not the package

I am trying to make a python package which I want to install using pip install . locally. The package name is listed in pip freeze but import <package> results in an error No module named <package>. Also the site-packages folder does only contain a dist-info folder. find_packages() is able to find packages. What am I missing?
import io
import os
import sys
from shutil import rmtree
from setuptools import find_packages, setup, Command
# Package meta-data.
NAME = '<package>'
DESCRIPTION = 'description'
URL = ''
EMAIL = 'email'
AUTHOR = 'name'
# What packages are required for this module to be executed?
REQUIRED = [
# 'requests', 'maya', 'records',
]
# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!
here = os.path.abspath(os.path.dirname(__file__))
# Where the magic happens:
setup(
name=NAME,
#version=about['__version__'],
description=DESCRIPTION,
# long_description=long_description,
author=AUTHOR,
author_email=EMAIL,
url=URL,
packages=find_packages(),
# If your package is a single module, use this instead of 'packages':
# py_modules=['mypackage'],
# entry_points={
# 'console_scripts': ['mycli=mymodule:cli'],
# },
install_requires=REQUIRED,
include_package_data=True,
license='MIT',
classifiers=[
# Trove classifiers
# Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
'License :: OSI Approved :: MIT License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy'
],
)

Since the question has become quite popular, here are the diagnosis steps to go through when you're missing files after installation. Imagine having an example project with the following structure:
root
├── spam
│ ├── __init__.py
│ ├── data.txt
│ ├── eggs.py
│ └── fizz
│ ├── __init__.py
│ └── buzz.py
├── bacon.py
└── setup.py
Now I run pip install ., check that the package is installed:
$ pip list
Package Version
---------- -------
mypkg 0.1
pip 19.0.1
setuptools 40.6.3
wheel 0.32.3
but see neither spam, nor spam/eggs.py nor bacon.py nor spam/fizz/buzz.py in the list of files belonging to the installed package:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
mypkg-0.1.dist-info/DESCRIPTION.rst
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/metadata.json
mypkg-0.1.dist-info/top_level.txt
So what to do now?
Diagnose by inspecting the wheel build log
Unless told not to do so, pip will always try to build a wheel file and install your package from it. We can inspect the log for the wheel build process if reinstalling in the verbose mode. First step is to uninstall the package:
$ pip uninstall -y mypkg
...
then install it again, but now with an additional argument:
$ pip install . -vvv
...
Now if I inspect the log:
$ pip install . -vvv | grep 'adding'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
I notice that no files from the spam directory or bacon.py are mentioned anywhere. This means they were simply not included in the wheel file and hence not installed by pip. The most common error sources are:
Missing packages: check the packages argument
Verify you have passed the packages argument to the setup function. Check that you have mentioned all of the packages that should be installed. Subpackages will not be collected automatically if only the parent package is mentioned! For example, in the setup script
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam']
)
spam will be installed, but not spam.fizz because it is a package itself and must be mentioned explicitly. Fixing it:
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam', 'spam.fizz']
)
If you have lots of packages, use setuptools.find_packages to automate the process:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages() # will return a list ['spam', 'spam.fizz']
)
In case you are missing a module:
Missing modules: check the py_modules argument
In the above examples, I will be missing bacon.py after installation since it doesn't belong to any package. I have to provide its module name in the separate argument py_modules:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
py_modules=['bacon']
)
Missing data files: check the package_data argument
I have all the source code files in place now, but the data.txt file is still not installed. Data files located under package directories should be added via the package_data argument. Fixing the above setup script:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
package_data={'spam': ['data.txt']},
py_modules=['bacon']
)
Don't be tempted to use the data_files argument. Place the data files under a package and configure package_data instead.
After fixing the setup script, verify the package files are in place after installation
If I now reinstall the package, I will notice all of the files are added to the wheel:
$ pip install . -vvv | grep 'adding'
adding 'bacon.py'
adding 'spam/__init__.py'
adding 'spam/data.txt'
adding 'spam/eggs.py'
adding 'spam/fizz/__init__.py'
adding 'spam/fizz/buzz.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
They will also be visible in the list of files belonging to mypkg:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
__pycache__/bacon.cpython-36.pyc
bacon.py
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/top_level.txt
spam/__init__.py
spam/__pycache__/__init__.cpython-36.pyc
spam/__pycache__/eggs.cpython-36.pyc
spam/data.txt
spam/eggs.py
spam/fizz/__init__.py
spam/fizz/__pycache__/__init__.cpython-36.pyc
spam/fizz/__pycache__/buzz.cpython-36.pyc
spam/fizz/buzz.py

For me, I noticed something weird if you do this:
# Not in the setup.py directory
python /path/to/folder/setup.py bdist_wheel
It will only install the .dist-info folder in your site-packages folder when you install the wheel.
However, if you do this:
cd /path/to/folder \
&& python setup.py bdist_wheel
The wheel will include all your files.

I had the same problem, and updating setuptools helped:
python3 -m pip install --upgrade pip setuptools wheel
After that, reinstall the package, and it should work fine :)

Make certain that your src files are in example_package_YOUR_USERNAME_HERE (this is the example package name that is used in the docs) and not in src. Errantly putting the files in src can have the effect described in the question.
Reference: https://packaging.python.org/en/latest/tutorials/packaging-projects/
The package should be set up like this:
packaging_tutorial/
└── src/
└── example_package_YOUR_USERNAME_HERE/
├── __init__.py
└── example.py

If you are on Windows 10+, one way you could make sure that you had all the correct installations was to click start in the bottom left-hand corner and search cmd.exe and right-click on "Command Prompt" (Make sure you choose "Run as Administrator"). Type "cd path to your Python 3.X installation". You can find this path in File Explorer (go to the folder where Python is installed) and then at the top. Copy this, and put it in where I wrote above path to your Python 3.X installation. Once you do that and click enter, type "python -m pip install package" (package signifies the package you would like to install). Your Python program should now work perfectly.