Short version:
How can I poetry install a package where one of the dependencies is a local tarball/zip file? It doesn't seem to work, yet it is shown in the poetry docs?
I can poetry install the package when the dependency is pulled from gitlab, but the install fails when I manually download the dependency from gitlab as a tarball and try to poetry install with the dependency in the tarball.
Long version:
I am trying to use poetry to install two packages that I have developed:
a base package called my_package
an extension called extension_of_my_package.
Both packages are in private repos in gitlab, and both have a pyproject.toml containing their dependency list. I can successfully poetry install the extended package (extension_of_my_package) when the base package my_package is downloaded from gitlab. i.e. the pyproject.toml file in extension_of_my_package has a tool.poetry.source section that gives the location of the my_package private repo on gitlab.
However, external users cannot access my private repo, so I need to ensure the
packages can be installed from tarballs (that I download from gitlab and give to the client).
To install extension_of_my_package I do this:
tar xzf extension_of_my_package.tgz
cd extension_of_my_package/python
and then edit the pyproject.toml, changing the dependency on my_package to point to
the local tarball:
my_package = { path = "/path/to/my_package.tgz"}
and then run poetry install. This fails with the error message:
> poetry install
Updating dependencies
Resolving dependencies... (9.3s)
TypeError
expected string or bytes-like object
at /home/user/.poetry/lib/poetry/_vendor/py3.8/poetry/core/utils/helpers.py:27 in canonicalize_name
23│ _canonicalize_regex = re.compile(r"[-_]+")
24│
25│
26│ def canonicalize_name(name): # type: (str) -> str
→ 27│ return _canonicalize_regex.sub("-", name).lower()
28│
29│
30│ def module_name(name): # type: (str) -> str
31│ return canonicalize_name(name).replace(".", "_").replace("-", "_")
According to the poetry docs it is possible to install from a local file:
[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = false }
# file
my-package = { path = "../my-package/dist/my-package-0.1.0.tar.gz" }
I also tried using my-package = { file = ... instead of my-package = { path = ..., but it didn't work either.
I tried adding a minimal setup.py file to my_package (see this post), but that didn't help.
I tried converting my_package (in tarball format) to a wheel. I can successfully poetry install when my package is in wheel format, but my_packages's dependencies are not installed. I can't see how to include the dependency info in the wheel. When I created the wheel I tried specifying the
dependency info in two ways:
in setup.cfg:
[metadata]
name = my_package
version = 0.1.0
description = My Package
license = Proprietary
[options]
packages = find:
install_requires =
matplotlib >=3.2.0
and
in setup.py"
from setuptools import setup
setup(
name=`my_package`,
version="0.1.0,
packages=['.my_package'],
install_requires=['matplotlib >= 3.2.0',]
)
To rule out any problem with my own package, I created a minimal test and tried to poetry install a publicly available package (tqdm) from its zip file (downloaded from github). It also fails. The pyproject.toml for this minimal test is:
[tool.poetry]
name = "tester"
version = "0.0.1"
description = "test package"
authors = [ "me" ]
packages = [
{ include = "tester" }
]
[tool.poetry.dependencies]
python = ">=3.7,<3.9"
tqdm = {file = "/home/user/tqdm-master.zip"}
and the error message is:
> poetry install
Updating dependencies
Resolving dependencies... (13.0s)
RuntimeError
Unable to determine package info from path: /home/user/tqdm-master.zip
at /home/user/.poetry/lib/poetry/puzzle/provider.py:251 in get_package_from_file
247│ package = PackageInfo.from_path(path=file_path).to_package(
248│ root_dir=file_path
249│ )
250│ except PackageInfoError:
→ 251│ raise RuntimeError(
252│ "Unable to determine package info from path: {}".format(file_path)
253│ )
254│
255│ return package
I am using poetry version 1.1.13.
I am open to any alternative approaches, so long as all the dependencies are checked.
Related
I find multiple answers for how to publish package using Python Poetry to CodeArtifact and this is quite simple. But now I try to add the published package poetry add sample-package and it does not work. Poetry error:
Could not find a matching version of package sample-package
With pip install it works. But not with Poetry.
My pyproject.toml specifies to you my CodeArtifact repo as default. No problem with this:
[[tool.poetry.source]]
name = "artifact"
url = "https://test-domain-1234.d.codeartifact.region.amazonaws.com/pypi/test-repo"
default = true
Did anyone figure out how to do it?
I found my mistake. In the package that I publish I need to specify repository without /simple at the end. But for project where I use the package from CodeArtifact the repository needs to end with /simple.
Example: Publish package config looks like:
[[tool.poetry.source]]
name = "artifact"
url = "https://test-domain-1234.d.codeartifact.region.amazonaws.com/pypi/test-repository/"
secondary=true
And the publish command is: poetry publish --build -r artifact
For project where I use my package sample-lib the config should be:
[[tool.poetry.source]]
name = "artifact-repo"
url = "https://test-domain-1234.d.codeartifact.region.amazonaws.com/pypi/test-repository/simple"
secondary=true
And then Poetry command is: poetry add sample-lib --source artifact-repo
I would like to configure pip to use a custom search path to install packages from a local folder, which are not hosted on PyPI. The goal is to be able to run
$ pip install --user my_non_published_package
And have it install said package from /home/myuser/projects/my_non_published_package.
I know that I can explicitely install it using
$ pip install --user /home/myuser/projects/my_non_published_package
but this is not what I want. I want pip to transparently resolve potential depencies of other packages and install them from /home/myuser/projects/<package> iff they are not hosted on PyPI. So, if I have a project foo having a setup.py containing
from setuptools import setup
setup(
...
install_requires=[
bar,
...
],
...
)
And package bar is not on PyPI, I want pip to install it from /home/myuser/projects/bar automatically.
Update
I found some information here, but the proposed solution does not work.
#! /usr/bin/env python3
"""Setup configuration."""
from pathlib import Path
from setuptools import setup
def local_pkg(name: str) -> str:
"""Returns a path to a local package."""
return f'{name} # file://{Path.cwd().parent / name}'
setup(
name='openimmodb',
use_scm_version={
"local_scheme": "node-and-timestamp"
},
setup_requires=['setuptools_scm'],
author='HOMEINFO - Digitale Informationssysteme GmbH',
author_email='<info at homeinfo dot de>',
maintainer='Richard Neumann',
maintainer_email='<r dot neumann at homeinfo period de>',
install_requires=[
local_pkg('filedb'),
local_pkg('mdb'),
local_pkg('openimmo'),
local_pkg('openimmolib'),
'peewee',
local_pkg('peeweeplus'),
'pyxb',
local_pkg('timelib'),
local_pkg('xmldom')
],
packages=[
'openimmodb',
'openimmodb.dom',
'openimmodb.dom.ausstattung',
'openimmodb.json',
'openimmodb.json.ausstattung',
'openimmodb.json.barrier_freeness',
'openimmodb.json.flaechen',
'openimmodb.json.immobilie',
'openimmodb.json.preise',
'openimmodb.mixins',
'openimmodb.orm'
],
entry_points={
'console_scripts': [
'oidbctl = openimmodb.oidbctl:main'
]
},
description='Relational OpenImmo database'
)
Results in:
...
Processing dependencies for openimmodb==0.1.dev987+g199a834.d20220110094301
Searching for xmldom# file:///home/neumann/Projekte/xmldom
Reading https://pypi.org/simple/xmldom/
Couldn't find index page for 'xmldom' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Even though it is there:
$ ls /home/neumann/Projekte/xmldom
LICENSE Makefile README.md setup.py venv xmldom.egg-info xmldom.py
Same with additional localhost:
Searching for xmldom# file://localhost/home/neumann/Projekte/xmldom
Reading https://pypi.org/simple/xmldom/
Couldn't find index page for 'xmldom' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
And with #egg=
Processing dependencies for openimmodb==0.1.dev987+g199a834.d20220110095236
Searching for xmldom# file://localhost/home/neumann/Projekte/xmldom#egg=xmldom
Reading https://pypi.org/simple/xmldom/
Couldn't find index page for 'xmldom' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
With the absolute path in install_requires I get:
$ python setup.py install
error in openimmodb setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Parse error at "'/home/ne'": Expected W:(abcd...)
I am trying to run poetry install command, but I get the following error:
[EnvCommandError]
Command ['pip', 'install', '-e', '<PROJECT_FOLDER_PATH>'] errored with
the following return code 1, and output:
Obtaining file:///<PROJECT_FOLDER_PATH>
ERROR: Package '<subfolder>' requires a different Python: 3.6.8 not in '>=3.7,<4.0'
Where my project directory containing .toml file is marked as <PROJECT_FOLDER> (PROJECT_FOLDER_PATH is, correspondingly, it's full path), and it contains <subfolder>.
Part of my toml file:
[tool.poetry]
name = "<PROJECT_FOLDER>"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.7"
It seems that poetry tries to install the project itself as a dependency, but for some reason that I don't understand it seems the conflicting Python version. I temporarily solved it by setting python = "^3.6", but now the issue is back, as I need some package which only accepts python = "^3.7".
Here's my setup.py
setup(
name='shipane_sdk',
version='1.0.0.a5',
# ...
data_files=[(os.path.join(os.path.expanduser('~'), '.shipane_sdk', 'config'), ['config/scheduler-example.ini'])],
# ...
)
Packing & Uploading commands:
python setup.py sdist
python setup.py bdist_wheel --universal
twine upload dist/*
Installing command:
pip install shipane_sdk
But, it doesn't install the config/scheduler-example.ini under ~/.shipane_sdk
The pip documents says:
setuptools allows absolute “data_files” paths, and pip honors them as
absolute, when installing from sdist. This is not true when installing
from wheel distributions. Wheels don’t support absolute paths, and
they end up being installed relative to “site-packages”. For
discussion see wheel Issue #92.
Do you know how to do installing from sdist?
There are multiple solutions to this problem and it is all very confusing how inconsistent the packaging tools work. Some time ago I found the following workaround worked for me best with sdist (note that it doesn't work with wheels!):
Instead of using data_files, attach the files to your package using MANIFEST.in, which in your case could look like this:
include config/scheduler-example.ini
Copy the files "manually" to chosen location using this snippet in setup.py:
if 'install' in sys.argv:
from pkg_resources import Requirement, resource_filename
import os
import shutil
# retrieve the temporary path where the package has been extracted to for installation
conf_path_temp = resource_filename(Requirement.parse(APP_NAME), "conf")
# if the config directory tree doesn't exist, create it
if not os.path.exists(CONFIG_PATH):
os.makedirs(CONFIG_PATH)
# copy every file from given location to the specified ``CONFIG_PATH``
for file_name in os.listdir(conf_path_temp):
file_path_full = os.path.join(conf_path_temp, file_name)
if os.path.isfile(file_path_full):
shutil.copy(file_path_full, CONFIG_PATH)
In my case "conf" was the subdirectory in the package that contained my data files and they were supposed to be installed into CONFIG_PATH which was something like /etc/APP_NAME
I am bitbaking a couple of python libraries and got this warning while adding second one of them:
WARNING: The recipe is trying to install files into a shared area when those files already exist. Those files are:
/home/ilya/beaglebone-dany/build/tmp/sysroots/beaglebone/usr/lib/python2.7/site-packages/site.py
/home/ilya/beaglebone-dany/build/tmp/sysroots/beaglebone/usr/lib/python2.7/site-packages/site.pyo
The libraries both use inherit distutils. So this is okay as far as bitbake goes, but when I tried to install the second package via opkg, I got this error:
# opkg install http://yocto.local:8080/python-requests_1.2.0-r0_armv7a-vfp-neon.ipk
Downloading http://yocto.local:8080/python-requests_1.2.0-r0_armv7a-vfp-neon.ipk.
Installing python-requests (1.2.0-r0) to root...
Configuring python-requests.
# opkg install http://yocto.local:8000/python-mylib_0.0.1-r0_armv7a-vfp-neon.ipk
Downloading http://yocto.local:8080/python-mylib_0.0.1-r0_armv7a-vfp-neon.ipk.
Installing python-mylib (0.0.1-r0) to root...
Collected errors:
* check_data_file_clashes: Package mylib-python wants to install file /usr/lib/python2.7/site-packages/site.py
But that file is already provided by package * python-requests
* check_data_file_clashes: Package mylib-python wants to install file /usr/lib/python2.7/site-packages/site.pyo
But that file is already provided by package * python-requests
* opkg_install_cmd: Cannot install package mylib-python.
Both recipes look just like so:
DESCRIPTION = "Requests: HTTP for Humans"
HOMEPAGE = "http://docs.python-requests.org/en/latest/"
SECTION = "devel/python"
LICENSE = "Apache-2.0"
DEPENDS = "python"
RDEPENDS_${PN} = "python-core"
PR = "r0"
SRC_URI = "git://github.com/kennethreitz/requests;protocol=git"
S = "${WORKDIR}/git/"
inherit distutils
#NOTE: I'm not 100% sure whether I still need to export these?
export BUILD_SYS
export HOST_SYS
export STAGING_INCDIR
export STAGING_LIBDIR
BBCLASSEXTEND = "native"
I have copied this from pycurl recipe, which also have had these lines that I removed:
do_install_append() {
rm -rf ${D}${datadir}/share
}
To get rid of the conflicting /usr/lib/python2.7/site-packages/site.py, one needs to avoid shipping this file by doing this:
do_install_append() {
rm -f ${D}${libdir}/python*/site-packages/site.py*
}
There had been another issue with the original version of recipe, the files it installed contained just .egg directory. I wasn't able to import the resulting package.
It turns out that using inherit setuptools instead of inherit distutils works.
I'm not a Python expert, but all setuptools class does is just this:
inherit distutils
DEPENDS += "python-setuptools-native"
DISTUTILS_INSTALL_ARGS = "--root=${D} \
--single-version-externally-managed \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
It turns out that some modules (e.g. PyBBIO) do not recognise --single-version-externally-managed, so then you have to use inherit distutils and you do get a working package.
Below is full recipe for the python-requests package which should soon be available upstream, in case you are intending to use it.
DESCRIPTION = "Requests: HTTP for Humans"
HOMEPAGE = "http://docs.python-requests.org/en/latest/"
SECTION = "devel/python"
LICENSE = "Apache-2.0"
#DEPENDS = "python-core"
RDEPENDS_${PN} = "python"
PR = "r0"
SRC_URI = "git://github.com/kennethreitz/requests;protocol=git"
S = "${WORKDIR}/git/"
inherit setuptools
# need to export these variables for python-config to work
export BUILD_SYS
export HOST_SYS
export STAGING_INCDIR
export STAGING_LIBDIR
BBCLASSEXTEND = "native"
do_install_append() {
rm -f ${D}${libdir}/python*/site-packages/site.py*
}