I'm trying to install pl_cross repository using poetry. In the README it's indicated that it must be installed running python setup.py install. When I try to install the repo adding it to my pyproject.toml file:
[tool.poetry]
name = "iq_segmentator"
version = "0.1.0"
description = "Speed up Deep Learning semantic segmentation projects."
authors = ["Jeremiah Poveda Martínez <jere#qubiotech.com>"]
[tool.poetry.dependencies]
python = "3.9"
wandb = "0.12.18"
nibabel = "4.0.1"
torch = "1.11.0"
pytorch-lightning = "1.6.4"
torchmetrics = "0.9.1"
monai = "0.9.0"
hydra-core = "1.2.0"
black = "22.3.0"
pytest = "7.1.2"
numpy = "1.22.4"
matplotlib = "3.5.2"
pl_cross = {git = "https://github.com/SkafteNicki/pl_cross.git", branch="master"}
sphinx = {version = "5.0.2", optional = true}
sphinxcontrib-napoleon = {version = "0.7", optional = true}
sphinx-rtd-theme = {version = "1.0.0", optional = true}
[tool.poetry.scripts]
pl_cross = "pl_closs:setup"
[tool.poetry.extras]
docs = ["sphinx", "sphinx-rtd-theme", "sphinxcontrib-napoleon"]
[tool.poetry.dev-dependencies]
mlflow = "^1.26.1"
pydra = "^0.18"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
I get the following error:
Updating dependencies
Resolving dependencies... (4.8s)
PackageInfoError
Unable to determine package info for path: /tmp/pypoetry-git-pl_crossah48_3rm
Fallback egg_info generation failed.
Command ['/tmp/tmpfc8t0g_3/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output:
Traceback (most recent call last):
File "/tmp/pypoetry-git-pl_crossah48_3rm/setup.py", line 15, in <module>
import pl_cross
File "/tmp/pypoetry-git-pl_crossah48_3rm/pl_cross/__init__.py", line 40, in <module>
from .datamodule import BaseKFoldDataModule, KFoldDataModule
File "/tmp/pypoetry-git-pl_crossah48_3rm/pl_cross/datamodule.py", line 5, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
at ~/.poetry/lib/poetry/inspection/info.py:500 in _pep517_metadata
496│ try:
497│ venv.run_python("setup.py", "egg_info")
498│ return cls.from_metadata(path)
499│ except EnvCommandError as fbe:
→ 500│ raise PackageInfoError(
501│ path, "Fallback egg_info generation failed.", fbe
502│ )
503│ finally:
504│ os.chdir(cwd.as_posix())
NOTE: I have torch installed. Could someone point out what is causing the error? Is is possible to isntall packages this way with poetry? Is this documented?
Poetry is not able to parse the needed metadata (package name, version and dependencies) from the setup.py. This is why a wheel package needs to be build where Poetry can obtain these information.
Building the wheel is done in a clean, temporary environment and all build-dependencies mentioned in the pyproject.toml according to PEP-518 are installed into this.
The problem with pl_cross is, that within the setup.py pl_cross is imported, which in return imports other modules and dependencies, which are not installed in the build environment.
Please contact the maintainer of this project and ask them to refactor it, so that the unnecessary imports doesn't happen.
Related
Short version:
How can I poetry install a package where one of the dependencies is a local tarball/zip file? It doesn't seem to work, yet it is shown in the poetry docs?
I can poetry install the package when the dependency is pulled from gitlab, but the install fails when I manually download the dependency from gitlab as a tarball and try to poetry install with the dependency in the tarball.
Long version:
I am trying to use poetry to install two packages that I have developed:
a base package called my_package
an extension called extension_of_my_package.
Both packages are in private repos in gitlab, and both have a pyproject.toml containing their dependency list. I can successfully poetry install the extended package (extension_of_my_package) when the base package my_package is downloaded from gitlab. i.e. the pyproject.toml file in extension_of_my_package has a tool.poetry.source section that gives the location of the my_package private repo on gitlab.
However, external users cannot access my private repo, so I need to ensure the
packages can be installed from tarballs (that I download from gitlab and give to the client).
To install extension_of_my_package I do this:
tar xzf extension_of_my_package.tgz
cd extension_of_my_package/python
and then edit the pyproject.toml, changing the dependency on my_package to point to
the local tarball:
my_package = { path = "/path/to/my_package.tgz"}
and then run poetry install. This fails with the error message:
> poetry install
Updating dependencies
Resolving dependencies... (9.3s)
TypeError
expected string or bytes-like object
at /home/user/.poetry/lib/poetry/_vendor/py3.8/poetry/core/utils/helpers.py:27 in canonicalize_name
23│ _canonicalize_regex = re.compile(r"[-_]+")
24│
25│
26│ def canonicalize_name(name): # type: (str) -> str
→ 27│ return _canonicalize_regex.sub("-", name).lower()
28│
29│
30│ def module_name(name): # type: (str) -> str
31│ return canonicalize_name(name).replace(".", "_").replace("-", "_")
According to the poetry docs it is possible to install from a local file:
[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = false }
# file
my-package = { path = "../my-package/dist/my-package-0.1.0.tar.gz" }
I also tried using my-package = { file = ... instead of my-package = { path = ..., but it didn't work either.
I tried adding a minimal setup.py file to my_package (see this post), but that didn't help.
I tried converting my_package (in tarball format) to a wheel. I can successfully poetry install when my package is in wheel format, but my_packages's dependencies are not installed. I can't see how to include the dependency info in the wheel. When I created the wheel I tried specifying the
dependency info in two ways:
in setup.cfg:
[metadata]
name = my_package
version = 0.1.0
description = My Package
license = Proprietary
[options]
packages = find:
install_requires =
matplotlib >=3.2.0
and
in setup.py"
from setuptools import setup
setup(
name=`my_package`,
version="0.1.0,
packages=['.my_package'],
install_requires=['matplotlib >= 3.2.0',]
)
To rule out any problem with my own package, I created a minimal test and tried to poetry install a publicly available package (tqdm) from its zip file (downloaded from github). It also fails. The pyproject.toml for this minimal test is:
[tool.poetry]
name = "tester"
version = "0.0.1"
description = "test package"
authors = [ "me" ]
packages = [
{ include = "tester" }
]
[tool.poetry.dependencies]
python = ">=3.7,<3.9"
tqdm = {file = "/home/user/tqdm-master.zip"}
and the error message is:
> poetry install
Updating dependencies
Resolving dependencies... (13.0s)
RuntimeError
Unable to determine package info from path: /home/user/tqdm-master.zip
at /home/user/.poetry/lib/poetry/puzzle/provider.py:251 in get_package_from_file
247│ package = PackageInfo.from_path(path=file_path).to_package(
248│ root_dir=file_path
249│ )
250│ except PackageInfoError:
→ 251│ raise RuntimeError(
252│ "Unable to determine package info from path: {}".format(file_path)
253│ )
254│
255│ return package
I am using poetry version 1.1.13.
I am open to any alternative approaches, so long as all the dependencies are checked.
I'm testing out the Natural Language API from Google but this line
from google.cloud import language_v1
is throwing the following error
ModuleNotFoundError: No module named 'google'
I am using Poetry and my pyproject.toml contains google-cloud
[tool.poetry]
name = "cocoon"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.9"
pandas = "^1.2.3"
app-store-scraper = "^0.3.5"
google-cloud-language = "^2.0.0"
google-api-python-client = "^2.0.2"
google-cloud = "^0.34.0"
fsspec = "^0.8.7"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
What I am following is based on these docs at Google.
There are similar questions (like this or this) but no solution has worked for me so far. I am not using a venv as outlined there since I am using Poetry. I have also double-checked on VSCode that the interpreter is correct.
I am trying to create a yocto recipe for scikit-learn package. It depends on scipy pacakge. I was able to successfully build the scipy package using : https://github.com/gpanders/meta-scipy.
When I run bitbake python3-scikit-learn, i am getting the below error:
ModuleNotFoundError: No module named 'scipy'
I am executing the commands in the below order.
Once I have cloned/copied the scipy recipes and the patches listed in the meta-scipy, i am running bitbake python3-scipy and the build was successful.
Then, I created a recipe file with the name python3-scikit-learn_0.23.2.bb and the contents are as below.
PYPI_PACKAGE = "scikit-learn"
LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=40ee42dc5a49f1617c5c78f16c50e065"
SRC_URI[sha256sum] = "20766f515e6cd6f954554387dfae705d93c7b544ec0e6c6a5d8e006f6f7ef480"
inherit pypi setuptools3
#DEPENDS = "${PYTHON_PN}-numpy-native ${PYTHON_PN}-numpy ${PYTHON_PN}-scipy ${PYTHON_PN}-joblib ${PYTHON_PN}"
DEPENDS = "${PYTHON_PN}-numpy-native ${PYTHON_PN}-numpy ${PYTHON_PN}-scipy ${PYTHON_PN}"
RDEPENDS_${PN} += "${PYTHON_PN}-numpy ${PYTHON_PN}-scipy"
When I run the bitbake python3-scikit-learn, i am getting this ModuleNotFoundError: No module named 'scipy'
Checked the path where the devshell python3 is looking (poky/build/tmp-glibc/work/aarch64-oe-linux/python3-scikit-learn/0.23.2-r0/recipe-sysroot-native/usr/lib/python3.8/site-packages), and i can only see the numpy package there, but scipy package is not there.
ls command output :
numpy
numpy-1.17.4-py3.8.egg-info
pkg_resources
__pycache__
README.txt
setuptools
setuptools-45.2.0-py3.8.egg-info
Can someone point me on how to include the python3-scipy package, so that it will be included/copied to the devshell. Or do I need to update/fix something else.
Appreciate any guidance on this.
you can also run:
bitbake -c devshell python3-scipy
and see exactly where the recipe is packaging everything into the rootfs. The rootfs is by default this:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n457
IMAGE_ROOTFS = "${WORKDIR}/rootfs"
so check out what python3-scipy puts into ${WORKDIR}/rootfs (${WORKDIR} is where you get thrown into after you execute devshell, so just cd into rootfs from there).
If python3-scipy puts it into somewhere that is not on the PATH, you can add that to your path.
you can see how python3-scipy looks for libraries:
https://github.com/gpanders/meta-scipy/blob/1c07824202af668ef1539c3de392cf737c5ba3fd/recipes-devtools/python/python3-scipy_1.5.3.bb#L29
# Tell Numpy to look in target sysroot site-packages directory for libraries
LDFLAGS_append = " -L${STAGING_LIBDIR}/${PYTHON_DIR}/site-packages/numpy/core/lib"
I am trying to run poetry install command, but I get the following error:
[EnvCommandError]
Command ['pip', 'install', '-e', '<PROJECT_FOLDER_PATH>'] errored with
the following return code 1, and output:
Obtaining file:///<PROJECT_FOLDER_PATH>
ERROR: Package '<subfolder>' requires a different Python: 3.6.8 not in '>=3.7,<4.0'
Where my project directory containing .toml file is marked as <PROJECT_FOLDER> (PROJECT_FOLDER_PATH is, correspondingly, it's full path), and it contains <subfolder>.
Part of my toml file:
[tool.poetry]
name = "<PROJECT_FOLDER>"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.7"
It seems that poetry tries to install the project itself as a dependency, but for some reason that I don't understand it seems the conflicting Python version. I temporarily solved it by setting python = "^3.6", but now the issue is back, as I need some package which only accepts python = "^3.7".
I followed this to add zc.recipe.testrunner to my buildout. I can run buildout successfully but when I run bin/test, I get:
ImportError: No module named testrunner
I have zope.testrunner-4.0.4-py2.4.egg in
/usr/local/lib/python2.4/site-packages
I also pinned
zope.testrunner = 4.0.4
zc.recipe.testruner = 1.4.0
zc.recipe.egg = 1.3.2
When I ran buildout, I used -vvv and I got:
...
Installing 'zc.recipe.testrunner'.
We have the distribution that satisfies 'zc.recipe.testrunner==1.4.0'.
Egg from site-packages: z3c.recipe.scripts 1.0.1
Egg from site-packages: zope.testrunner 4.0.4
Egg from site-packages: zope.interface 3.8.0
Egg from site-packages: zope.exceptions 3.7.1
...
We have the distribution that satisfies 'zope.testrunner==4.0.4'.
Egg from site-packages: zope.testrunner 4.0.4
Adding required 'zope.interface'
required by zope.testrunner 4.0.4.
We have a develop egg: zope.interface 0.0
Adding required 'zope.exceptions'
required by zope.testrunner 4.0.4.
We have a develop egg: zope.exceptions 0.0
...
Why is it I get an ImportError? Is zope.testrunner not installed correctly?
Edit:
This is the relevant part in my buildout:
[buildout]
...
parts =
...
test
[test]
recipe = zc.recipe.testrunner
defaults = ['--auto-color', '--auto-progress']
eggs =
my.product
This is the content in bin/test:
#!/usr/local/bin/python2.4 -S
import sys
sys.path[0:0] = [
'/home/jil/mySandbox/myTrunk/parts/test/site-packages',
]
import os
path = sys.path[0]
if os.environ.get('PYTHONPATH'):
path = os.pathsep.join([path, os.environ['PYTHONPATH']])
os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ.get('PYTHONPATH', '')
os.environ['PYTHONPATH'] = path
import site # imports custom buildout-generated site.py
import os
sys.argv[0] = os.path.abspath(sys.argv[0])
os.chdir('/home/jil/mySandbox/myTrunk/parts/test/working-directory')
import zope.testrunner
if __name__ == '__main__':
zope.testrunner.run((['--auto-color', '--auto-progress']) + [
'--test-path', '/home/jil/mySandbox/myTrunk/src/my.product',
])
This is the error after running bin/test:
Traceback (most recent call last):
File "/home/jil/mySandbox/myTrunk/bin/test", line 20, in ?
import zope.testrunner
ImportError: No module named testrunner
I had the same problem. At least in my case, the cause was mixing dependencies already installed in 'site-packages' and dependencies installed by buildout in 'eggs': zope.deprecation and zope.interface were already in my 'site-packages' directory and thus were not re-installed by buildout. The path manipulation on the 'bin/test' executable seemed to import the 'zope' package from 'site-packages', without the 'testrunner' subpackage.
Try removing all zope.* packages from 'site-packages' and re-run buildout, or use 'include-site-packages = false' in the '[buildout]' section of your buildout.cfg. The first solution worked for me.