I'm using azure devops for building a python project where we have a requirements.txt that references private git repositories.
These repos are under the same github account that our devops account has access to, but are unable to get them using pip.
I notice that the 'Get sources' step adds an extra header to git commands:
[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --tags --prune --progress --no-recurse-submodules origin
Is there any way that we can get this header into other steps to allow us to use the same authentication for private pip installation or otherwise access private git repositories that devops already has access to?
I do not want to use personal access tokens, or deploy keys as it just another layer of security to be maintained and managed. Azure devops already has authentication to these repositories and we should be able to use this.
Related
I am trying to clone a GCP Cloud Source Repository (CSR) using Cloud Build.
when trying to clone the Repo in my Dockerfile, I get this error:
fatal: could not read Username for 'https://source.developers.google.com': No such device or address
This appears to be an authentication error with git and CSR.
What I've tried: I have found https://cloud.google.com/build/docs/build-config-file-schema#network and this github issue https://github.com/GoogleCloudPlatform/cloud-builders/issues/343. I may not have put them in the right place in the config file though.
I'm aware that I could just use an SSH key, but I'd like to use the "inherited" authentication if that's possible. By "inherited" I mean use the cloud build service account.
If anyone can help with the specific issue, that would be great. However if you can enlighten me as to how authentication works from the Build to other GCP services that would be welcome.
-ps I'm actually installing via pip (just like the person in the GitHub issue linked above), but it needs to clone the repo first, and that's where the error is.
You can use the https://source.developers.google.com URL with a PAT(Personal Access Token) in the Dockerfile.
For that you need to generate a PAT.
After generating the token, use it in the dockerfile as follows:
FROM gcr.io/cloud-builders/git
RUN git clone https://<PAT>#source.developers.google.com/p/<PROJECT_ID>/r/<REPO_NAME>
I'm trying to install a private python package in the Google Cloud composer environment. I usually install the package using a personal access token. That works with usual pip.
pip install git+https://$TOKEN#github.com/org/repo.git#main
works as expected. However, trying to use this in Cloud Composer tells me the following:
PyPI package name must follow the format of 'identifier' specified in PEP-508. (https://peps.python.org/pep-0508/)
I'm not sure what I should do here. Does anyone have some experience with this?
If the repo hosting your private Python packages accepts external calls, you can generate a pip.conf file with params to access to this repo :
[global]
extra-index-url = https://username:password#yourrepo.com/simple
Then you have to copy this file to Cloud Composer bucket : {composer_bucket_name}/config/pip/pip.conf
You can also host your Python private packages in GCP Artifact registry, then generate a pip.conf file allowed to access to the created Python repo in Artifact registry and then copy this file to the Composer bucket : {composer_bucket_name}/config/pip/pip.conf
After this you can add your private Python Packages in the PyPi packages list from Cloud Composer, like public packages :
You can also check this link from official documentation : https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies
I need to delete a python package from a private python package index (a.k.a. repository) using the command line. This repo is on artifactory, but I am not able to use the artifactory portal UI to do this.
The package was originally uploaded using twine. However, it looks like there is no delete functionality in twine.
Twine was able to succeed at the upload despite being agnostic of it being an artifactory repo... so I assume there is some kind of standardized pypi-type api...?
(This question is similar but different to How to remove a package from Pypi , because it is asking specifically about a private repo and asking specifically about a CLI solution)
As you have mentioned that the package has been uploaded to the Artifactory's repository using "twine", I assume the package currently exists in a PyPI local repository of the Artifactory instance.
Since you are looking for an option to delete this artifact from the repository via the command line, please check if this REST API call to delete the artifact is an option for you.
I am sharing a sample command here for your reference.
curl -u <USERNAME>:<PASSWORD> -X DELETE "http://ARTIFACTORY_HOST:ARTIFACTORY_PORT/artifactory/admin-pypi-remote-cache/bd/"
admin-pypi-remote is my repository name and bd is the target folder/package of the deletion task.
if you are using a local pypi server, to delete an uploaded artifact use the following command:
curl -u <USERNAME>:<PASSWORD> --form ":action=remove_pkg" --form "name=<PACKAGE_NAME>" --form "version=<VERSION>" <PYPI_SERVER_URL>
I am using PyPi to clone a github repository, however it a private repo. So I also need to authorise it with my token. I am unable to find any example of how to pass token with the clone request.
import git
git.Git('/local/file/path').clone('git#github.com:sample-repo.git', token=(mytoken))
This gives an error
"GitCommandError: Cmd('git') failed due to: exit code(129)"
cmdline: git clone --token=mytoken git#github.com:sample-repo.git
stderr: 'error: unknown option token=mytoken'
It works fine without the token when I try to clone a public repository. So the only issue here is how to pass the token to the above request. Is that possible or is there any other way to authorise git clone in a python script? My objective here is to automate a process to clone a github repository, generate some files using some API calls, add and commit those files to the repository, all within the same python script.
Thanks to the comments, I was able to clone my repository using the https url instead of the ssh url and it worked without the need for token.
import git
git.Git('/local/file/path').clone('https://github.com/sample-repo')
Previously I worked on a Node.js project I ran a private npm registry and used PayPal's Kappa to proxy between this and the public npm registry. I also used an npmrc file in each project to define the url of the registry. Now when developers checkout the projects and run npm install the request will go to Kappa which first looks in the local registry and if it doesn't find the matching module it forwards the request to the public registry.
Is there anyway to achieve this with Python projects. Ideally I'd like the developers to simple check out a service and use pip install -r requirements.txt without knowing if the dependencies are coming from public or private PyPi.