I would like to be able to install python packages to openshift, but those packages live in my private repositories, on bitbucket.
How can I create a SSH key for Openshift, and how do I make Openshift use it when installing packages? (after adding the corresponding public key to bitbucket as a Deploy Key)
What I've tried:
I used ssh-keygen to create a key on ~/.openshift_ssh/. It was created, but I'm not sure it is being used.
I also tried adding the publick key on <jenkins_dir>/app-root/data/.ssh/jenkins_id_rsa.pub, but the result is always the same. On the jenkins console output of the buildjob:
Doing git clone from ssh://git#bitbucket.org/jpimentel/zed.git to /tmp/easy_install-FpEKam/zed.git
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Is there anything that can be done?
So, at this time OpenShift does not offer a simple mechanism to do this. I would urge developers to look at https://gondor.io/support/dependencies/ for an effective solution to the problem.
That said, I was finally able to find an acceptable (at least, for me) workaround that works on both scalable and non scalable apps, with the following procedure:
create a deploy/ directory in the repository
put a copy of your private deploy key in said directory
create a bash script deploy/wrapper.sh that will run ssh with the provided key:
#!/bin/sh
ssh -o StrictHostKeyChecking=no -i $OPENSHIFT_REPO_DIR/deploy/id_deploy $#
note the option passed to disable host key check; cloning will fail without it.
install dependencies in the build hook (.openshift/action_hooks/build). In my case I added something like
echo "Cloning private repo..."
source $VIRTUAL_ENV/bin/activate
GIT_SSH=$OPENSHIFT_REPO_DIR/deploy/wrapper.sh pip install git+ssh://git#bitbucket.org/team/reponame.git#egg=reponame
commit everything and push it to openshift.
profit!
If you want to deploy your custom python modules then recommended way is to create a libs directory in the application source code root and push them to your application git repository. OpenShift will automatically pick your modules.
Related
I need to delete a python package from a private python package index (a.k.a. repository) using the command line. This repo is on artifactory, but I am not able to use the artifactory portal UI to do this.
The package was originally uploaded using twine. However, it looks like there is no delete functionality in twine.
Twine was able to succeed at the upload despite being agnostic of it being an artifactory repo... so I assume there is some kind of standardized pypi-type api...?
(This question is similar but different to How to remove a package from Pypi , because it is asking specifically about a private repo and asking specifically about a CLI solution)
As you have mentioned that the package has been uploaded to the Artifactory's repository using "twine", I assume the package currently exists in a PyPI local repository of the Artifactory instance.
Since you are looking for an option to delete this artifact from the repository via the command line, please check if this REST API call to delete the artifact is an option for you.
I am sharing a sample command here for your reference.
curl -u <USERNAME>:<PASSWORD> -X DELETE "http://ARTIFACTORY_HOST:ARTIFACTORY_PORT/artifactory/admin-pypi-remote-cache/bd/"
admin-pypi-remote is my repository name and bd is the target folder/package of the deletion task.
if you are using a local pypi server, to delete an uploaded artifact use the following command:
curl -u <USERNAME>:<PASSWORD> --form ":action=remove_pkg" --form "name=<PACKAGE_NAME>" --form "version=<VERSION>" <PYPI_SERVER_URL>
To run InstaBot locally, you just clone the repo, install the requirements.txt, put in your login credentials in example.py, and run python example.py. I do not know how this translates to OpenShift.
Let's say you push your code to your own GitHub repo with the login credentials in environment variables (in an git ignored file). You can set environment variables on the OpenShift dashboard, but where's the part where you specify python example.py?
For OpenShift, if example.py is a self contained Python web application, then you would need to rename it as app.py, or add a .s2i/environment file to your repo and in it add:
APP_FILE=example.py
The script should then ensure it is listening on all interfaces, ie., 0.0.0.0 and not just localhost. It also needs to use port 8080.
With that done, you can then use Python S2I builder process in OpenShift to deploy it. The packages listed in requirements.txt will be automatically installed for you.
if not familiar with OpenShift, you might consider reading:
https://www.openshift.com/deploying-to-openshift/
It is a free download.
For details on the Python S2I builder and what environment variables you can set to customise it, see:
https://github.com/sclorg/s2i-python-container/tree/master/3.6
I'm trying to install private python-based git repos from a requirements.txt into a docker container such that they are easily editable during development.
For example, I have a Django project which contains a Dockerfile that allows building that project inside of a docker container. (It might look something like this https://github.com/JoeJasinski/docker-django-demo/blob/master/Dockerfile).
Now, say that project has a requirements.txt file that pulls in code from a private repos as follows.
django=1.11.2
-e git+git#github.com:myorg/my-private-project.git#egg=my_private_project
-e git+ssh://git#git.example.com/second-private-project#mytag#egg=second_private_project
-e git+https://github.com/myorg/third-private-project#egg=third_private_project
Ideally, I'd make it so I can edit both my main project, and the dependent repos without having to re-build the docker container each time. The Dockerfile "ADD . dest/" command makes it possible for the main project to be edited in place, but I'm having difficulty finding a good solution for installing these private repositories.
Normally (outside of Docker), the pip -e flag makes repos editable in place, which is great since I can edit and commit to them like any other repo.
However, inside of Docker, the container doesn't have access to the ssh private key needed to download the private repos (and this is probably a good thing, so we don't build the key into the docker images).
One thought I had is to download the private repos outside of the container, prior to building. Then somehow those repos would be "ADD"ed to the Docker container at build time and then individually added to the PYTHONPATH (maybe during runtime?). However, I feel like I'm over-complicating the situation.
Any suggestions as to a good, simple (Pythonic) way to install private python-based git repositories into a container so that it's easy to develop on both the main project and dependent repositories?
Here is the scenario I am dealing with:
I WANT to/HAVE setup CircleCI build for my project with unit tests etc.
In this project I use another one of my libraries which needs to be installed on the build container in CirleCi, otherwise my tests are failing.
I need to find a way to either:
pull git repository of external reference and install it
Or download it as zip
Or some other way ?
Happy to add more explanation if needed.
From the section Using Resources External to Your Repository:
CircleCI supports git submodule, and has advanced SSH key management to let you access multiple repositories from a single test suite. From your project’s Project Settings > Checkout SSH keys page, you can add a “user key” with one-click, allowing you access code from multiple repositories in your test suite. Git submodules can be easily set up in your circle.yml file (see example 1).
CircleCI’s VMs are connected to the internet. You can download dependencies directly while setting up your project, using curl or wget.
(Or just using git clone without submodules.)
I am trying to make a python script which would check for changes in my git local working folder, and automatically push them to the online repo. Currently only using git manually to do it. I want to know, what would a script require to do this without manual intervention.
The commands I'd type in my shell are:
#for checking the status, and determining if there are untracked files
git status
#if there are untracked files...add them
git add .
#add my commit message
git commit -m "7/8/2012 3:25am"
#push it to my online repo
git push origin master
#check if changes came on remote
git diff origin/master
#merge my repo with origin
git merge origin/master
When doing the git push, you'd always have to enter username/password. I know that git has a way around this which involves making ssh keys and all. But I am assuming there is some way GitPython is doing it. I mean we can pass username/password through code, or go with the former. So what are my options regarding authentication when I am using GitPython?
Edit: There are apps which actually generate the ssh keys, for e.g. github's windows application. How is the windows app doing this? My assumption is that there is surely some git api for it...
If you are authenticating using SSH keys, then use ssh-agent to load your key once and then you can keep using the key without having to provide the password all the time.
Alternatively, you could simply generate a key without a password, if you don’t care about your key security.
I have looked up the code to be sure, there is nothing that you can define a username/password combination for communication.
This has to be it because ssh does not give you the ability to provide password beforehand, it intentionally asks for user prompt. The only was is to use ssh keys for automation.
However, if you really want to bend your limits. There's open source app for non-interaction ssh communication without using ssh keys: http://sourceforge.net/projects/sshpass/
You compile & install this and direct a communication protocol like ssh:// to this app, it may work. However, I don't think that you should; just use keys, they're great =)