I find multiple answers for how to publish package using Python Poetry to CodeArtifact and this is quite simple. But now I try to add the published package poetry add sample-package and it does not work. Poetry error:
Could not find a matching version of package sample-package
With pip install it works. But not with Poetry.
My pyproject.toml specifies to you my CodeArtifact repo as default. No problem with this:
[[tool.poetry.source]]
name = "artifact"
url = "https://test-domain-1234.d.codeartifact.region.amazonaws.com/pypi/test-repo"
default = true
Did anyone figure out how to do it?
I found my mistake. In the package that I publish I need to specify repository without /simple at the end. But for project where I use the package from CodeArtifact the repository needs to end with /simple.
Example: Publish package config looks like:
[[tool.poetry.source]]
name = "artifact"
url = "https://test-domain-1234.d.codeartifact.region.amazonaws.com/pypi/test-repository/"
secondary=true
And the publish command is: poetry publish --build -r artifact
For project where I use my package sample-lib the config should be:
[[tool.poetry.source]]
name = "artifact-repo"
url = "https://test-domain-1234.d.codeartifact.region.amazonaws.com/pypi/test-repository/simple"
secondary=true
And then Poetry command is: poetry add sample-lib --source artifact-repo
Related
I’m publishing a Python package.
It has dependencies a, b, c. It also has pytest as a dependency, defined in the group.dev as per poetry’s documentation.
However, I couldn’t find a conclusive answer for the following question:
When some installs my library with:
pip install my_library
Is pytest (defined in any group other than the main group, in this case dev) also installed? That would be very undesirable.
You can mention the dev dependency like this. It will not install the pytest. Reference
# rp_poetry/pyproject.toml (Excerpt)
[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.26.0"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
black = "^21.9b0"
poetry is may go to change the label of dev-dependencies in the future.
So I'm having a problem for quite some time which I cannot get solved. Basically I took over a project which uses Poetry for package managing (It is a Django project). Adding packages with 'poetry add' and then installing them via 'poetry install' all works fine locally (I use a Docker container). But when pushing the changes to my server and then running 'poetry install' it says the packages are already installed. But when running the Django application, I get an internal server error saying the package doesn't exist. An example is given with the 'openpyxl' package.
pyproject.toml
...
[tool.poetry.dependencies]
openpyxl = "^3.0.10"
...
poetry.lock
...
[[package]]
name = "openpyxl"
version = "3.0.10"
description = "A Python library to read/write Excel 2010 xlsx/xlsm files"
category = "main"
optional = false
python-versions = ">=3.6"
[package.dependencies]
openpyxl = {version = ">=2.6.0", optional = true, markers = "extra == \"xlsx\""}
[package.extras]
all = ["markuppy", "odfpy", "openpyxl (>=2.6.0)", "pandas", "pyyaml", "tabulate", "xlrd", "xlwt"]
cli = ["tabulate"]
html = ["markuppy"]
ods = ["odfpy"]
pandas = ["pandas"]
xls = ["xlrd", "xlwt"]
xlsx = ["openpyxl (>=2.6.0)"]
yaml = ["pyyaml"]
openpyxl = [
{file = "openpyxl-3.0.10-py2.py3-none-any.whl", hash = "sha256:0ab6d25d01799f97a9464630abacbb34aafecdcaa0ef3cba6d6b3499867d0355"},
{file = "openpyxl-3.0.10.tar.gz", hash = "sha256:e47805627aebcf860edb4edf7987b1309c1b3632f3750538ed962bbcc3bd7449"},
]
...
error:
Anyone experienced with Poetry who can help me with this?
I'm making an educated guess on few ideas you can try (without knowing more about the problem/context). For starters:
when running / starting your app, are you using poetry run... / poetry run python... ? I'm listing this one first, since it seems like the py environment isn't accessing the installed libs. would like to know the cmd "when running the Django application"
on the server, you can try to remove the lock file and re-run poetry lock, poetry install to have a 'fresh start'. The lock file you provided is the one from the server?
how do you use Docker, is this only on local? do you use Docker on the server as a running container? You can try dockerizing your app, and as part of the Dockerfile you copy pyproject.toml/poetry.lock files have docker RUN poetry cmds to install your deps.
again, these are shots in the dark. Some areas that can use some more understanding/detail are (a) how you start up Django app and (b) how you use Docker. If you're looking at point #3, I can elaborate on a solution for you. I've worked with quite a bit of docker+poetry apps/services.
Short version:
How can I poetry install a package where one of the dependencies is a local tarball/zip file? It doesn't seem to work, yet it is shown in the poetry docs?
I can poetry install the package when the dependency is pulled from gitlab, but the install fails when I manually download the dependency from gitlab as a tarball and try to poetry install with the dependency in the tarball.
Long version:
I am trying to use poetry to install two packages that I have developed:
a base package called my_package
an extension called extension_of_my_package.
Both packages are in private repos in gitlab, and both have a pyproject.toml containing their dependency list. I can successfully poetry install the extended package (extension_of_my_package) when the base package my_package is downloaded from gitlab. i.e. the pyproject.toml file in extension_of_my_package has a tool.poetry.source section that gives the location of the my_package private repo on gitlab.
However, external users cannot access my private repo, so I need to ensure the
packages can be installed from tarballs (that I download from gitlab and give to the client).
To install extension_of_my_package I do this:
tar xzf extension_of_my_package.tgz
cd extension_of_my_package/python
and then edit the pyproject.toml, changing the dependency on my_package to point to
the local tarball:
my_package = { path = "/path/to/my_package.tgz"}
and then run poetry install. This fails with the error message:
> poetry install
Updating dependencies
Resolving dependencies... (9.3s)
TypeError
expected string or bytes-like object
at /home/user/.poetry/lib/poetry/_vendor/py3.8/poetry/core/utils/helpers.py:27 in canonicalize_name
23│ _canonicalize_regex = re.compile(r"[-_]+")
24│
25│
26│ def canonicalize_name(name): # type: (str) -> str
→ 27│ return _canonicalize_regex.sub("-", name).lower()
28│
29│
30│ def module_name(name): # type: (str) -> str
31│ return canonicalize_name(name).replace(".", "_").replace("-", "_")
According to the poetry docs it is possible to install from a local file:
[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = false }
# file
my-package = { path = "../my-package/dist/my-package-0.1.0.tar.gz" }
I also tried using my-package = { file = ... instead of my-package = { path = ..., but it didn't work either.
I tried adding a minimal setup.py file to my_package (see this post), but that didn't help.
I tried converting my_package (in tarball format) to a wheel. I can successfully poetry install when my package is in wheel format, but my_packages's dependencies are not installed. I can't see how to include the dependency info in the wheel. When I created the wheel I tried specifying the
dependency info in two ways:
in setup.cfg:
[metadata]
name = my_package
version = 0.1.0
description = My Package
license = Proprietary
[options]
packages = find:
install_requires =
matplotlib >=3.2.0
and
in setup.py"
from setuptools import setup
setup(
name=`my_package`,
version="0.1.0,
packages=['.my_package'],
install_requires=['matplotlib >= 3.2.0',]
)
To rule out any problem with my own package, I created a minimal test and tried to poetry install a publicly available package (tqdm) from its zip file (downloaded from github). It also fails. The pyproject.toml for this minimal test is:
[tool.poetry]
name = "tester"
version = "0.0.1"
description = "test package"
authors = [ "me" ]
packages = [
{ include = "tester" }
]
[tool.poetry.dependencies]
python = ">=3.7,<3.9"
tqdm = {file = "/home/user/tqdm-master.zip"}
and the error message is:
> poetry install
Updating dependencies
Resolving dependencies... (13.0s)
RuntimeError
Unable to determine package info from path: /home/user/tqdm-master.zip
at /home/user/.poetry/lib/poetry/puzzle/provider.py:251 in get_package_from_file
247│ package = PackageInfo.from_path(path=file_path).to_package(
248│ root_dir=file_path
249│ )
250│ except PackageInfoError:
→ 251│ raise RuntimeError(
252│ "Unable to determine package info from path: {}".format(file_path)
253│ )
254│
255│ return package
I am using poetry version 1.1.13.
I am open to any alternative approaches, so long as all the dependencies are checked.
I am writing a python application which dependents on Scrapy module. It works fine locally but failed when I run it from aws lambda test console. My python project has a requirements.txt file with below dependency:
scrapy==1.6.0
I packaged all dependencies by following this link: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html. And also, I put my source code *.py at the root level of in the zip file. My package script can be found https://github.com/zhaoyi0113/quote-datalake/blob/master/bin/deploy.sh.
It basically does two things, first run command pip install -r requirements.txt -t dist to download all dependencies to dist directory. second, copy app python source code to dist directory.
The deployment is done via terraform and below is the configuration file.
provider "aws" {
profile = "default"
region = "ap-southeast-2"
}
variable "runtime" {
default = "python3.6"
}
data "archive_file" "zipit" {
type = "zip"
source_dir = "crawler/dist"
output_path = "crawler/dist/deploy.zip"
}
resource "aws_lambda_function" "test_lambda" {
filename = "crawler/dist/deploy.zip"
function_name = "quote-crawler"
role = "arn:aws:iam::773592622512:role/LambdaRole"
handler = "handler.handler"
source_code_hash = "${data.archive_file.zipit.output_base64sha256}"
runtime = "${var.runtime}"
}
It zip the directory and upload the file to lambda.
I found I get the runtime error in lambda Unable to import module 'handler': cannot import name 'etree' when there is a statement import scrapy. I didn't use etree in my code so I believe there is something used by scrapy.
My source code can be found at https://github.com/zhaoyi0113/quote-datalake/tree/master/crawler. There are only two simple python files.
It works fine if I run them locally. The error only appears in lambda. Is there a different way to package scrapy to lambda?
Based on the communication with Tim, the issue is caused by incompatible library versions between local and lambda.
The easiest way to resolve this issue is to use the docker image lambci/lambda to build a package with the command:
$ docker run -v $(pwd):/outputs -it --rm lambci/lambda:build-python3.6 pip install scrapy -t /outputs/
You need to provide the entire dependency tree, scrapy also has a set of dependencies (and they may also have dependencies).
The easiest way to download all the required dependencies is to use pip
$ pip -t packages/ install scrapy
This will download scrapy and all its dependencies into the folder packages.
Scrapy has lxml and pyOpenSSL as dependencies that include compiled components. Unless they are statically compiled they will likely require that the c-libraries they require are also installed on the lambda VM.
From the lxml documentation it requires:
libxml2 version 2.9.2 or later.
libxslt version 1.1.27 or later.
We recommend libxslt 1.1.28 or later.
Maybe try adding installation of these to your deploy script. You should be able to use (I'm making a guess at the package names) yum -y install libxml2 libxslt
Another good idea is to test your scripts on an Amazon Linux EC2 instance as this is close to the environment that Lambda executes in.
I installed a package from git hub:
pip install -e git+http://github.com/un33k/django-uuslug.git#egg=django-uuslug
Then I did:
pip freeze > req.txt
I get:
django-uuslug==0.1
Now if I do a pip install -r req.txt, I get a package not found error, which due to the fact that django-uuslug is not on pypi.
Why is freeze not remembering the full path as it was given during the install?
I had the same issue. I believe it's a problem whenever the packages are in a subdirectory(e.g. src). Here's the patch that fixed it for me.
--- a/setup.py
+++ b/setup.py
## -11,13 +11,9 ## setup(
license = 'BSD',
description = "MAC address model and form fields for Django apps.",
long_description = read('README.rst'),
-
author = 'Ryan Nowakowski',
author_email = 'me#example.com',
-
- packages = find_packages('src'),
- package_dir = {'': 'src'},
-
+ packages = ['macaddress'],
install_requires = ['setuptools'],
requires = ['netaddr'],
#tests_requires = ['django'],
I fixed it, don't know how, but I had to change the setup.py
pip install -e git+http://github.com/un33k/django-uuslug.git#egg=django-uuslug
If you find similar issue, and find yourself on this question, just look at the setup.py in the above package. Perhaps you can tell me how I fixed it. I just moved things around a bit.