Develop Reference

Python3 with Bazel 0.5.4 - hard-coded to python executable?

I'm maintaining a project that is using Bazel 0.5.4, and it needs to run on an environment where python does not exist. I'd like to point the application to using python3.
This issue is similar to Bazel 0.26.1 use Python3 on py_test, but the version of rules_python mentioned in that ticket is so new that it's not compatible with bazel 0.5.4
GOAL: Issue python commands with /usr/bin/python3, not python
As-is, the application fails when it's unable to find python
bazel build completes without an issue, but bazel test continues to look for python, and fail.
bazel test ... --test_output=all --python_top=//sandbox_app:python-3.8.10
...
/usr/bin/env: 'python': No such file or directory
================================================================================
INFO: Elapsed time: 0.658s, Critical Path: 0.10s
INFO: Build completed, 1 test FAILED, 2 total actions
//sandbox_app:sandbox_test FAILED in 0.1s
This didn't work
The following configuration worked for a similar user, described here:
https://groups.google.com/g/bazel-discuss/c/nVQ48R94S_8
But even after a bazel clean, bazel still continues to invoke python when running the tests.
WORKSPACE
git_repository(
name = "io_bazel_rules_go",
remote = "https://github.com/bazelbuild/rules_go.git",
tag = "0.5.4",
)
BUILD.bazel
package(default_visibility = ["//visibility:public"])
py_test(
name = "sandbox_test",
srcs = ["sandbox_test.py"],
default_python_version = "PY3",
srcs_version = "PY3",
)
py_runtime(
name = "python-3.8.10",
files = [],
interpreter_path = "/usr/bin/python3",
)
Version details:
$ bazel info release
release 0.5.4
It would minimize disruption to the application if this can be done without upgrading bazel.
sandbox_test.py
#!/usr/bin/python3
import unittest
class SandboxTest(unittest.TestCase):
def testRunSandbox(self):
# Valid for Python 3, syntax error for python2
print(print("Hello, Python!"))
if __name__ == '__main__':
unittest.main()
Possibly related issues
https://github.com/bazelbuild/bazel/issues/4815
https://github.com/bazelbuild/bazel/issues/200
Rules spec:
https://bazel.build/reference/be/python
Similar issues:
How do I select the runtime in bazel for Python and pip?
Bazel, python3 and python interpreter options
Getting Bazel to run using Python3 (syntax error)

Install a python repo from setup.py in poetry

I'm trying to install pl_cross repository using poetry. In the README it's indicated that it must be installed running python setup.py install. When I try to install the repo adding it to my pyproject.toml file:
[tool.poetry]
name = "iq_segmentator"
version = "0.1.0"
description = "Speed up Deep Learning semantic segmentation projects."
authors = ["Jeremiah Poveda Martínez <jere#qubiotech.com>"]
[tool.poetry.dependencies]
python = "3.9"
wandb = "0.12.18"
nibabel = "4.0.1"
torch = "1.11.0"
pytorch-lightning = "1.6.4"
torchmetrics = "0.9.1"
monai = "0.9.0"
hydra-core = "1.2.0"
black = "22.3.0"
pytest = "7.1.2"
numpy = "1.22.4"
matplotlib = "3.5.2"
pl_cross = {git = "https://github.com/SkafteNicki/pl_cross.git", branch="master"}
sphinx = {version = "5.0.2", optional = true}
sphinxcontrib-napoleon = {version = "0.7", optional = true}
sphinx-rtd-theme = {version = "1.0.0", optional = true}
[tool.poetry.scripts]
pl_cross = "pl_closs:setup"
[tool.poetry.extras]
docs = ["sphinx", "sphinx-rtd-theme", "sphinxcontrib-napoleon"]
[tool.poetry.dev-dependencies]
mlflow = "^1.26.1"
pydra = "^0.18"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
I get the following error:
Updating dependencies
Resolving dependencies... (4.8s)
PackageInfoError
Unable to determine package info for path: /tmp/pypoetry-git-pl_crossah48_3rm
Fallback egg_info generation failed.
Command ['/tmp/tmpfc8t0g_3/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output:
Traceback (most recent call last):
File "/tmp/pypoetry-git-pl_crossah48_3rm/setup.py", line 15, in <module>
import pl_cross
File "/tmp/pypoetry-git-pl_crossah48_3rm/pl_cross/__init__.py", line 40, in <module>
from .datamodule import BaseKFoldDataModule, KFoldDataModule
File "/tmp/pypoetry-git-pl_crossah48_3rm/pl_cross/datamodule.py", line 5, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
at ~/.poetry/lib/poetry/inspection/info.py:500 in _pep517_metadata
496│ try:
497│ venv.run_python("setup.py", "egg_info")
498│ return cls.from_metadata(path)
499│ except EnvCommandError as fbe:
→ 500│ raise PackageInfoError(
501│ path, "Fallback egg_info generation failed.", fbe
502│ )
503│ finally:
504│ os.chdir(cwd.as_posix())
NOTE: I have torch installed. Could someone point out what is causing the error? Is is possible to isntall packages this way with poetry? Is this documented?

Poetry is not able to parse the needed metadata (package name, version and dependencies) from the setup.py. This is why a wheel package needs to be build where Poetry can obtain these information.
Building the wheel is done in a clean, temporary environment and all build-dependencies mentioned in the pyproject.toml according to PEP-518 are installed into this.
The problem with pl_cross is, that within the setup.py pl_cross is imported, which in return imports other modules and dependencies, which are not installed in the build environment.
Please contact the maintainer of this project and ask them to refactor it, so that the unnecessary imports doesn't happen.

Poetry won't install dependency in a tarball/zip file

Short version:
How can I poetry install a package where one of the dependencies is a local tarball/zip file? It doesn't seem to work, yet it is shown in the poetry docs?
I can poetry install the package when the dependency is pulled from gitlab, but the install fails when I manually download the dependency from gitlab as a tarball and try to poetry install with the dependency in the tarball.
Long version:
I am trying to use poetry to install two packages that I have developed:
a base package called my_package
an extension called extension_of_my_package.
Both packages are in private repos in gitlab, and both have a pyproject.toml containing their dependency list. I can successfully poetry install the extended package (extension_of_my_package) when the base package my_package is downloaded from gitlab. i.e. the pyproject.toml file in extension_of_my_package has a tool.poetry.source section that gives the location of the my_package private repo on gitlab.
However, external users cannot access my private repo, so I need to ensure the
packages can be installed from tarballs (that I download from gitlab and give to the client).
To install extension_of_my_package I do this:
tar xzf extension_of_my_package.tgz
cd extension_of_my_package/python
and then edit the pyproject.toml, changing the dependency on my_package to point to
the local tarball:
my_package = { path = "/path/to/my_package.tgz"}
and then run poetry install. This fails with the error message:
> poetry install
Updating dependencies
Resolving dependencies... (9.3s)
TypeError
expected string or bytes-like object
at /home/user/.poetry/lib/poetry/_vendor/py3.8/poetry/core/utils/helpers.py:27 in canonicalize_name
23│ _canonicalize_regex = re.compile(r"[-_]+")
24│
25│
26│ def canonicalize_name(name): # type: (str) -> str
→ 27│ return _canonicalize_regex.sub("-", name).lower()
28│
29│
30│ def module_name(name): # type: (str) -> str
31│ return canonicalize_name(name).replace(".", "_").replace("-", "_")
According to the poetry docs it is possible to install from a local file:
[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = false }
# file
my-package = { path = "../my-package/dist/my-package-0.1.0.tar.gz" }
I also tried using my-package = { file = ... instead of my-package = { path = ..., but it didn't work either.
I tried adding a minimal setup.py file to my_package (see this post), but that didn't help.
I tried converting my_package (in tarball format) to a wheel. I can successfully poetry install when my package is in wheel format, but my_packages's dependencies are not installed. I can't see how to include the dependency info in the wheel. When I created the wheel I tried specifying the
dependency info in two ways:
in setup.cfg:
[metadata]
name = my_package
version = 0.1.0
description = My Package
license = Proprietary
[options]
packages = find:
install_requires =
matplotlib >=3.2.0
and
in setup.py"
from setuptools import setup
setup(
name=`my_package`,
version="0.1.0,
packages=['.my_package'],
install_requires=['matplotlib >= 3.2.0',]
)
To rule out any problem with my own package, I created a minimal test and tried to poetry install a publicly available package (tqdm) from its zip file (downloaded from github). It also fails. The pyproject.toml for this minimal test is:
[tool.poetry]
name = "tester"
version = "0.0.1"
description = "test package"
authors = [ "me" ]
packages = [
{ include = "tester" }
]
[tool.poetry.dependencies]
python = ">=3.7,<3.9"
tqdm = {file = "/home/user/tqdm-master.zip"}
and the error message is:
> poetry install
Updating dependencies
Resolving dependencies... (13.0s)
RuntimeError
Unable to determine package info from path: /home/user/tqdm-master.zip
at /home/user/.poetry/lib/poetry/puzzle/provider.py:251 in get_package_from_file
247│ package = PackageInfo.from_path(path=file_path).to_package(
248│ root_dir=file_path
249│ )
250│ except PackageInfoError:
→ 251│ raise RuntimeError(
252│ "Unable to determine package info from path: {}".format(file_path)
253│ )
254│
255│ return package
I am using poetry version 1.1.13.
I am open to any alternative approaches, so long as all the dependencies are checked.

Yocto recipe for scikit-learn

I am trying to create a yocto recipe for scikit-learn package. It depends on scipy pacakge. I was able to successfully build the scipy package using : https://github.com/gpanders/meta-scipy.
When I run bitbake python3-scikit-learn, i am getting the below error:
ModuleNotFoundError: No module named 'scipy'
I am executing the commands in the below order.
Once I have cloned/copied the scipy recipes and the patches listed in the meta-scipy, i am running bitbake python3-scipy and the build was successful.
Then, I created a recipe file with the name python3-scikit-learn_0.23.2.bb and the contents are as below.
PYPI_PACKAGE = "scikit-learn"
LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=40ee42dc5a49f1617c5c78f16c50e065"
SRC_URI[sha256sum] = "20766f515e6cd6f954554387dfae705d93c7b544ec0e6c6a5d8e006f6f7ef480"
inherit pypi setuptools3
#DEPENDS = "${PYTHON_PN}-numpy-native ${PYTHON_PN}-numpy ${PYTHON_PN}-scipy ${PYTHON_PN}-joblib ${PYTHON_PN}"
DEPENDS = "${PYTHON_PN}-numpy-native ${PYTHON_PN}-numpy ${PYTHON_PN}-scipy ${PYTHON_PN}"
RDEPENDS_${PN} += "${PYTHON_PN}-numpy ${PYTHON_PN}-scipy"
When I run the bitbake python3-scikit-learn, i am getting this ModuleNotFoundError: No module named 'scipy'
Checked the path where the devshell python3 is looking (poky/build/tmp-glibc/work/aarch64-oe-linux/python3-scikit-learn/0.23.2-r0/recipe-sysroot-native/usr/lib/python3.8/site-packages), and i can only see the numpy package there, but scipy package is not there.
ls command output :
numpy
numpy-1.17.4-py3.8.egg-info
pkg_resources
__pycache__
README.txt
setuptools
setuptools-45.2.0-py3.8.egg-info
Can someone point me on how to include the python3-scipy package, so that it will be included/copied to the devshell. Or do I need to update/fix something else.
Appreciate any guidance on this.

you can also run:
bitbake -c devshell python3-scipy
and see exactly where the recipe is packaging everything into the rootfs. The rootfs is by default this:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n457
IMAGE_ROOTFS = "${WORKDIR}/rootfs"
so check out what python3-scipy puts into ${WORKDIR}/rootfs (${WORKDIR} is where you get thrown into after you execute devshell, so just cd into rootfs from there).
If python3-scipy puts it into somewhere that is not on the PATH, you can add that to your path.
you can see how python3-scipy looks for libraries:
https://github.com/gpanders/meta-scipy/blob/1c07824202af668ef1539c3de392cf737c5ba3fd/recipes-devtools/python/python3-scipy_1.5.3.bb#L29
# Tell Numpy to look in target sysroot site-packages directory for libraries
LDFLAGS_append = " -L${STAGING_LIBDIR}/${PYTHON_DIR}/site-packages/numpy/core/lib"

Building python packages

I am bitbaking a couple of python libraries and got this warning while adding second one of them:
WARNING: The recipe is trying to install files into a shared area when those files already exist. Those files are:
/home/ilya/beaglebone-dany/build/tmp/sysroots/beaglebone/usr/lib/python2.7/site-packages/site.py
/home/ilya/beaglebone-dany/build/tmp/sysroots/beaglebone/usr/lib/python2.7/site-packages/site.pyo
The libraries both use inherit distutils. So this is okay as far as bitbake goes, but when I tried to install the second package via opkg, I got this error:
# opkg install http://yocto.local:8080/python-requests_1.2.0-r0_armv7a-vfp-neon.ipk
Downloading http://yocto.local:8080/python-requests_1.2.0-r0_armv7a-vfp-neon.ipk.
Installing python-requests (1.2.0-r0) to root...
Configuring python-requests.
# opkg install http://yocto.local:8000/python-mylib_0.0.1-r0_armv7a-vfp-neon.ipk
Downloading http://yocto.local:8080/python-mylib_0.0.1-r0_armv7a-vfp-neon.ipk.
Installing python-mylib (0.0.1-r0) to root...
Collected errors:
* check_data_file_clashes: Package mylib-python wants to install file /usr/lib/python2.7/site-packages/site.py
But that file is already provided by package * python-requests
* check_data_file_clashes: Package mylib-python wants to install file /usr/lib/python2.7/site-packages/site.pyo
But that file is already provided by package * python-requests
* opkg_install_cmd: Cannot install package mylib-python.
Both recipes look just like so:
DESCRIPTION = "Requests: HTTP for Humans"
HOMEPAGE = "http://docs.python-requests.org/en/latest/"
SECTION = "devel/python"
LICENSE = "Apache-2.0"
DEPENDS = "python"
RDEPENDS_${PN} = "python-core"
PR = "r0"
SRC_URI = "git://github.com/kennethreitz/requests;protocol=git"
S = "${WORKDIR}/git/"
inherit distutils
#NOTE: I'm not 100% sure whether I still need to export these?
export BUILD_SYS
export HOST_SYS
export STAGING_INCDIR
export STAGING_LIBDIR
BBCLASSEXTEND = "native"
I have copied this from pycurl recipe, which also have had these lines that I removed:
do_install_append() {
rm -rf ${D}${datadir}/share
}

To get rid of the conflicting /usr/lib/python2.7/site-packages/site.py, one needs to avoid shipping this file by doing this:
do_install_append() {
rm -f ${D}${libdir}/python*/site-packages/site.py*
}
There had been another issue with the original version of recipe, the files it installed contained just .egg directory. I wasn't able to import the resulting package.
It turns out that using inherit setuptools instead of inherit distutils works.
I'm not a Python expert, but all setuptools class does is just this:
inherit distutils
DEPENDS += "python-setuptools-native"
DISTUTILS_INSTALL_ARGS = "--root=${D} \
--single-version-externally-managed \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
It turns out that some modules (e.g. PyBBIO) do not recognise --single-version-externally-managed, so then you have to use inherit distutils and you do get a working package.
Below is full recipe for the python-requests package which should soon be available upstream, in case you are intending to use it.
DESCRIPTION = "Requests: HTTP for Humans"
HOMEPAGE = "http://docs.python-requests.org/en/latest/"
SECTION = "devel/python"
LICENSE = "Apache-2.0"
#DEPENDS = "python-core"
RDEPENDS_${PN} = "python"
PR = "r0"
SRC_URI = "git://github.com/kennethreitz/requests;protocol=git"
S = "${WORKDIR}/git/"
inherit setuptools
# need to export these variables for python-config to work
export BUILD_SYS
export HOST_SYS
export STAGING_INCDIR
export STAGING_LIBDIR
BBCLASSEXTEND = "native"
do_install_append() {
rm -f ${D}${libdir}/python*/site-packages/site.py*
}