I want to install multiple private Github repositories as python packages into my repository using poetry. This works fine when running poetry install locally as I have my public SSH key added to Github which allows poetry to access the private repos. The problem is that I want to install these same private packages in my CI/CD pipeline and for that I need to add one deploy key per repo. The correct deploy key needs to be used towards the correct repository and for that to work, I need to setup a config with some aliases on the following format (I haven't actually gotten so far as to know whether this will actually work or not):
// /.ssh/config
Host repo-1.github.com
IdentityFile ~/.ssh/repo-1-deploy-key
Host repo-2.github.com
IdentityFile ~/.ssh/repo-2-deploy-key
Where repo-1 and repo-2 are the names of the private repositories I need to install. When running locally the pyproject.toml packages need to be setup in the following format:
// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git#github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git#github.com/equinor/repo-2.git" }
...
As this will allow the developer to install the packages without any configuration (given that they have access). For the CI/CD pipeline however, the URLs need to match the alias in the SSH config file and therefore it needs to look something like this:
// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git#repo-1.github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git#repo-2.github.com/equinor/repo-2.git" }
...
Now, where I seem to be stuck at is on how I can include two different git paths in the same pyproject file? I tried the following:
//pyproject.toml
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git#repo-1.github.com/equinor/repo-1.git", optional=true }
repo-2 = { git = "ssh://git#repo-2.github.com/equinor/repo-2.git", optional=true }
[tool.poetry.dev-dependencies]
repo-1 = { git = "ssh://git#repo-1.github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git#repo-2.github.com/equinor/repo-2.git" }
[tool.poetry.extras]
cicd_modules = ["repo-1", "repo-2"]
So that I could run poetry install locally and it would use the dev dependencies and poetry install --no-dev --extras cicd_modules in the CI/CD pipeline to use the aliased paths. Sadly, this gives me a CalledProcessError as it seems like poetry tries to install the optional package despite the optional flag set to true.
What am I doing wrong here, am I using the optional flag incorrectly somehow? Is there a better way to solve this? In summary I just want to be able to install multiple private repositories as packages using poetry and Github deploy keys in a CI/CD pipeline without breaking local install behaviour, if it's in this way or another better way, I don't really have a strong opinion on.
Related
I have a python package I want to test and build using Jenkins and tox.
However, it depends on a package in our private artifactory. Although poetry picks up on it correctly, the moment tox kicks in (and uses pip over poetry to install dependencies), it can't find the package.
My poetry lock file has
reference = "snakepit"
type = "legacy"
url = "https://our private artifactory
My Jenkinsfile sets up POETRY_HTTP_BASIC_PYPI_USERNAME and POETRY_HTTP_BASIC_PYPI_PASSWORD which allows poetry to install without the url in the lock file having my credentials.
After some searching it seemed like I needed to extend legacy_tox_ini in the pyproject.toml. This now looks like:
legacy_tox_ini = """
[tox]
isolated_build = true
envlist = py36,py37,py38
indexserver =
default = https://pypi.python.org/simple
ourartifactory = https://{env:POETRY_HTTP_BASIC_PYPI_USERNAME}:{env:POETRY_HTTP_BASIC_PYPI_PASSWORD}#our artifactory url
[testenv]
deps =
pytest
pytest-cov
setenv =
SOME_ENV={env:SOME_ENV}
commands =
pytest tests
I also tested adding :ourartifactory:problempackage to deps here, but it doesn't help either.
How can I make pip recognize the private artifactory in this setting?
In the end the whole indexserver and deps was unnecessary, and a simple
envVar(key: 'PIP_CONFIG_FILE', value: "/pypi-artifactory/pip.conf")
in the Jenkinsfile sufficed. Of course this requires your docker image to have the correct pip.conf.
This question is about how to manage access for private gitlab python packages with nested dependencies on other private gitlab packages. This assumes all the access is via direct git repository patterns and not a private package repository.
package-a is in a private gitlab repository and it depends on package-b, which depends on package-c and they are also in a private gitlab repository.
package-a has a pyproject.toml like this:
[tool.poetry]
name = "package-a"
repository = "https://gitlab.com/org/package_a.git"
[tool.poetry.dependencies]
python = "^3.6"
package-b = {git = "ssh://git#gitlab.com/org/package_b.git", tag = "0.1.0"}
package-b has a pyproject.toml like this:
[tool.poetry]
name = "package-b"
repository = "https://gitlab.com/org/package_b.git"
[tool.poetry.dependencies]
python = "^3.6"
package-c = {git = "ssh://git#gitlab.com/org/package_c.git", tag = "0.1.0"}
Any user with the right org membership on gitlab and an ssh-key can use poetry to install package-a and it's dependency on package-b and then it's dependency on package-c, all into a python venv on a development laptop. The ssh protocol access also works for docker builds (with experimental features for ssh mounts).
However, the same projects with private dependencies are not installed in gitlab-CI runners, because they lack ssh access. (Is there any secure way to enable that?)
Assuming the gitlab-CI runners must use an access token to clone the private gitlab repositories, a sed script is applied to the pyproject.toml file for project-a, so the gitlab-CI runner can clone package-b to discover that it depends on package-c; the sed script changes the ssh to https access for project-b by editing the project-a dependency spec in pyproject.toml, i.e.
sed -i -e 's#ssh://git#gitlab.com/org#https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com/org#g' pyproject.toml
The CI_JOB_TOKEN is an environment variable provided by the gitlab-CI runner. It is securely managed by the gitlab-CI runner. So, gitlab-CI runner can now clone the project-b repository somewhere. The same sed trick might work if it could be applied to this project-b repository somewhere, but this is now in the hands of poetry and it cannot be touched. So project-b has a git+ssh dependency on project-c and the gitlab-CI runner fails to install project-c because it has no git+ssh credentials to clone it.
Thus, the private package dependency chain works for development and docker build on a laptop with git+ssh access, but it's all broken on gitlab-CI. What's the better practice to manage this private package access across all these build environments?
These snippets are based on:
https://docs.gitlab.com/ee/ci/ssh_keys/
https://gitlab.com/gitlab-examples/ssh-private-key
https://gitlab.com/gitlab-examples/ssh-private-key/issues/7
https://gitlab.com/gitlab-examples/ssh-private-key/issues/1
ssh-keygen -o -t rsa -b 4096 -C "git#gitlab.com"
# output to something like ~/.ssh/gitlab_ci_rsa
# do not add any passphrase
# once created, copy the private key to the clipboard, e.g.
cat ~/.ssh/gitlab_ci_rsa | base64 -w0 > tmp.txt
xclip -sel clip < tmp.txt
The public key is used as a private deploy key,
which is enabled from a project settings page, e.g.
https://gitlab.com/org/project-a/settings/repository
https://gitlab.com/org/project-b/settings/repository
https://gitlab.com/org/project-c/settings/repository
The private key is pasted into the gitlab-CI variable SSH_PRIVATE_KEY
and gitlab should be able to mask it (when it is base64 encoded). Then the .gitlab-ci.yml file can add this private key to the ssh-agent using:
before_script:
- apt-get update -y -qq && apt-get install -y -qq git make openssh-client
- eval $(ssh-agent -s)
## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 --decode)
## Create the SSH directory and give it the right permissions
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
The gitlab documentation does not use base64 encoding, but it's essential to copying all of the private key into the variable and that avoids a prompt for a passphrase from ssh-add.
We are using Pipenv and Pipfiles for managing our Python package requirements. We use a private GitLab server to manage our code, including as a source for packages, eg via:
ourpkg = {ref = "master", git = "ssh://git#gitlab.company:/ourpkg.git"}
However, the code is deployed to a separate Prod environment which has internet access (and so can pull down pypi packges), but cannot access our gitlab server.
Our first proposed solution was to mirror the package repos on the Prod environment. However, while Pipenv allows environment variables in a package index source it doesn't allow them in git urls in packages, so we can't define a different base URL in each environment that way.
Our second proposed solution was to host the packages on a private package index, use that as a source on the Prod environment. Then specify prod links under [packages], and the gitlab git urls under [dev-packages].
[[source]]
url = "https://pypi.prodenvironment/
name = "prodindex"
[packages]
ourpkg = {version = "*", index = "prodindex"}
[dev-packages]
ourpkg = {ref = "master", git = "ssh://git#gitlab.company:/ourpkg.git"}
However, this fails on installing in dev as the [packages] requirements take precedence over [dev-packages], and Pipenv does not allow deploy-only requirements.
How can we specify a git repo source for a package in dev, and a different (index or git repo) source for a package in production?
I have a project being built on Jenkins using the multibranch pipeline plugin. I am using the declarative pipeline syntax and my Jenkinsfile looks something like this:
pipeline {
agent { label 'blah' }
options {
timeout(time: 2, unit: 'HOURS')
buildDiscarder(logRotator(numToKeepStr: '5'))
}
triggers { pollSCM('H/5 * * * *') }
stages {
stage('Prepare') {
steps {
sh '''
echo "Building environment"
python3 -m venv venv && \
pip install git+ssh://git#my_private_repo.git
'''
}
}
}
}
When the build is run on the Jenkins box the build fails and when I check the console output it is failing on the pip install command with the error:
Permission denied (publickey).
fatal: Could not read from remote repository.
I am guessing that I need to set the required ssh key into jenkins build environment, but am not sure how to do this.
You need to install the SSH Agent plugin and use it to wrap the actions in the steps directive in order to be able to pull from a private repository. You enable the SSH Agent with the sshagent directive, where you need to pass in an argument representing the hash for a valid key with read permissions to the git repository. The key needs to be available in the global credentials view of Jenkins (Jenkins -> Credentials [on the left-hand side menu], search for the ID field of the right key), e.g.:
stage('Prepare') {
steps {
sshagent(['<hash_for_your_key>']) {
echo "Building environment"
sh "python3.5 -m venv venv"
sh "venv/bin/python3.5 venv/bin/pip install git+ssh://git#my_private_repo.git
}
}
N.B.: Because the actions under the steps directive are executed as subprocesses, you'll need to call explicitly the executable files from the virtual environment, using long syntax.
When I tried to set one of the services into AWS Elastic Beanstalk, the problem appeared. One of our python pip dependencies set up on private repository was causing errors during deployment process, since it was not accessible for pip process. Bellow I present description how we solved this issue.
It is worth to mention that there are other solutions that bind SSH keys used during deployment, to application project git repository. I find them a bit dirty, so I would like to share this one that allows to keep SSH keys separated in S3 bucket, separated from application git repository.
Works on: "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4" for dependency from private bitbucket repository. it may be modified for github, etc.
Add you private dependency to pip requirements.txt:
-e git+git#bitbucket.org:account_name/dependency-name.git#egg=dependency-name
Generate public & private SSH keys (HOWTO can be found elsewhere) so you have id_rsa (private) and id_rsa.pub (public) key files.
On bitbucket.org project tab, find settings and go under "Deployment keys". Use form to set your public SSH key there.
Upload both generated keys (just private should be enough) on S3 bucket using amazon AWS console:
bucket-with-keys/bitbucket/:
- id_rsa
- id_rsa.pub
Where bucket-with-keys/bitbucket - is [BUCKET_NAME]/[PATH].
Add packages file to your project: project_name/.ebextensions/packages.config
packages:
yum:
git: []
Add configuration file to your project: project_name/.ebextensions/03-pip-install-from-bitbucket.config:
files:
"/root/.ssh/config":
owner: root
group: root
mode: "000600"
content: |
Host *
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
"/root/.ssh/known_hosts":
owner: root
group: root
mode: "000644"
content: |
#
# paste output of `ssh-keyscan -H github.com` here
#
commands:
01-command:
command: aws s3 cp --recursive s3://bucket-with-keys/bitbucket/ /root/.ssh
02-command:
command: chmod 600 /root/.ssh/id_rsa
If you modify this file, keep in mind that:
commands are being executed in alphabetical order so "01-" starts before "02-" even if you would switch its order.
file name also follows that rule.
Go to AWS console IAM (Identity & Access Management) "policies" tab and find AmazonS3FullAccess on a list and attache it for aws-elasticbeanstalk-ec2-role.
If you cannot find policy you can create one like below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
Look for "Create and Attach Your First Customer Managed Policy" Tutorial if needed. This policy is quite open and it is advised to create one that would be more narrow...
aws-elasticbeanstalk-ec2-role is role created by default by Elastic Beanstalk but you can use your own as long as it is set with CLI tool eb config under: "IamInstanceProfile: aws-elasticbeanstalk-ec2-role"
Now you can create your environmant with eb CLI tool:
eb create project_name-prod-env \
--instance_type t2.micro \
--tier webserver \
--region eu-west-1 \
--cname project_name-prod-env \
--keyname identifier-of-ssh-key-accessed-from-console-here \
--platform "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4"
Should work now!
If sth goes wrong you may debug if SSH files got to its place:
eb ssh
sudo su
ls -la /root/.ssh/
And check logs on AWS console or directly on instance:
eb ssh
sudo su
less /var/log/eb-activity.log
Try manually execute commands from project_name/.ebextensions/03-pip-install-from-bitbucket.config as a root user bu in the way they appear in log file, use switches to get more verbose output.
I cannot comment so I'm answering the question. smentek's answer is very detailed and solves the issue. The only missing part is that you need to add the git package to the config:
packages:
yum:
git-all: ""
files:
"/root/.ssh/config":
# ...
Git isn't installed on Amazon Linux by default.