Git clone private repo in AWS ECS - python

I'm trying to download all private repos in my organisation.
I have a script that I want to run once a day using Fargate.
The problem I'm experiencing when running it is below:
Warning: Permanently added the RSA host key for IP address '192.30.253.113' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
I understand the error and in my dockerfile I add an ssh key to the image:
FROM python:3.6
RUN mkdir /backup
WORKDIR /backup
ADD . /backup/
RUN mkdir /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
RUN chmod 400 /root/.ssh/id_rsa
RUN python3 -m pip install -r requirements.txt
This is the script where I try to download all repos and upload them to S3 bucket:
TOKEN = os.environ['TOKEN']
DATE = str(date.today())
def archive(zipname, directory):
return shutil.make_archive(zipname, 'zip', root_dir=directory,
base_dir=None)
def assume_role(role_to_assume, duration=900):
sts_client = boto3.client('sts')
assumed_role = sts_client.assume_role(
RoleArn=role_to_assume,
RoleSessionName='session',
DurationSeconds=duration
)
credentials = assumed_role['Credentials']
return (credentials['AccessKeyId'], credentials['SecretAccessKey'],
credentials['SessionToken'])
def upload_to_s3(key, file_name, access_role):
access_key, secret_key, session_token = assume_role(access_role)
s3 = boto3.resource(
's3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
aws_session_token=session_token
)
s3.Bucket('zego-github-backup').put_object(
Key=key,
Body=file_name
)
print('Uploaded')
def login_github():
g = Github(TOKEN)
org = g.get_organization("Organisation").get_repos()
role = "arn:aws:iam::7893729191287:role/Github_backup"
for repo in org:
repo_name = repo.name
key = f"{repo_name} {DATE}.zip"
ssh_url = repo.ssh_url
os.system(f"GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no\" git clone --depth 1 {ssh_url}")
archive(f"{repo_name} {DATE}", repo_name)
archived_file = open(key, 'rb')
upload_to_s3(key, archived_file, role)
shutil.rmtree(repo_name)
os.remove(f"{repo_name} {DATE}.zip")
print("Done")
login_github()
What am I doing wrong? Or am I missing some steps?

Not sure if I missed anything from your script but I didn't see you starting the ssh-agent anywhere and then adding the key to it.
From GitHub's guide,
$ eval "$(ssh-agent -s)"
$ ssh-add /root/.ssh/id_rsa
Hopefully that helps!

There is a very simple to use python tool that automatically backs up organisations' repositories in .zip format by saving public and private repositories and all their branches. If you want to do your backups every x amount of time, all you have to do is run the tool from a crontab on your server. It works with the Github API, if you have python on your AWS instance, the tool will be very useful : https://github.com/BuyWithCrypto/OneGitBackup

Related

Passing AWS credentials to Python container

I am building Python container for the first time using VS Code and WSL2. Here is my sample Python code. It runs fine in VS interactive mode because it is picking up my default AWS credentials.
import boto3
s3BucketName = 'test-log-files'
s3 = boto3.resource('s3')
def s3move():
try:
s3.Object(s3BucketName, "destination/Changes.xlsx").copy_from(CopySource=(s3BucketName + "/source/Changes.xlsx"))
s3.Object(s3BucketName,"source/Changes.xlsx").delete()
print("Transfer Complete")
except:
print("Transfer failed")
if __name__ == "__main__":
s3move()
The Dockerfile built by VS Code:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim-buster
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
RUN pip install boto3
WORKDIR /app
COPY . /app
# Switching to a non-root user, please refer to https://aka.ms/vscode-docker-python-user-rights
RUN useradd appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "S3MoveFiles/S3MoveFiles.py"]
I would like to test this using docker container and seems like I have to pass the AWS credentials to the container. While there are other ways and probably more secure ways, I wanted to test the method by mounting the volume in a docker command as an argument.
docker run -v ~/.aws/credentials:/appuser/home/.aws/credentials:ro image_id
I get the "Transfer failed" message in the Terminal window in VS Code. What am I doing wrong here? Checked several articles but couldn't find any hints. I am not logged in as root.

Cannot push to multiple repositories in GitHub from local [duplicate]

Trying to work on both my actual "work" repos, and my repos on GitHub, from my computer.
The work account was set up first, and everything works flawlessly.
My account, however, cannot seem to push to my repo, which is set up under a different account/email.
I've tried copying my work key up to my account, but that throws an error because of course a key can only be attached to one account.
How can I push/pull to and from both accounts with their respective GitHub credentials?
All you need to do is configure your SSH setup with multiple SSH keypairs.
This link is easy to follow (Thanks Eric):
http://code.tutsplus.com/tutorials/quick-tip-how-to-work-with-github-and-multiple-accounts--net-22574
Generating SSH keys (Win/msysgit):
https://help.github.com/articles/generating-an-ssh-key/
Relevant steps from the first link:
Generate an SSH-key:
ssh-keygen -t ed25519 -C "john#doe.example.com"
Follow the prompts and decide a name, e.g. id_ed25519_example_company.
Copy the SSH public-key to GitHub from ~/.ssh/id_ed25519_doe_company.pub and tell ssh about the key:
ssh-add ~/.ssh/id_ed25519_doe_company
Create a config file in ~/.ssh with the following contents:
Host github-doe-company
HostName github.com
User git
IdentityFile ~/.ssh/id_ed25519_doe_company
Add your remote:
git remote add origin git#github-doe-company:username/repo.git
or change using:
git remote set-url origin git#github-doe-company:username/repo.git
Also, if you're working with multiple repositories using different personas, you need to make sure that your individual repositories have the user settings overridden accordingly:
Setting user name, email and GitHub token – Overriding settings for individual repos
https://help.github.com/articles/setting-your-commit-email-address-in-git/
Note:
Some of you may require different emails to be used for different repositories, from git 2.13 you can set the email on a directory basis by editing the global config file found at: ~/.gitconfig using conditionals like so:
[user]
name = Default Name
email = defaultemail#example.com
[includeIf "gitdir:~/work/"]
path = ~/work/.gitconfig
And then your work-specific config ~/work/.gitconfig would look like this:
[user]
name = Pavan Kataria
email = pavan.kataria#example.com
Thank you #alexg for informing me of this in the comments.
Use HTTPS:
change remote url to https:
git remote set-url origin https://USERNAME#github.com/USERNAME/PROJECTNAME.git
and you are good to go:
git push
To ensure that the commits appear as performed by USERNAME, one can setup the user.name and user.email for this project, too:
git config user.name USERNAME
git config user.email USERNAME#example.com
Getting into shape
To manage a git repo under a separate github/bitbucket/whatever account, you simply need to generate a new SSH key.
But before we can start pushing/pulling repos with your second identity, we gotta get you into shape – Let's assume your system is setup with a typical id_rsa and id_rsa.pub key pair. Right now your tree ~/.ssh looks like this
$ tree ~/.ssh
/Users/you/.ssh
├── known_hosts
├── id_rsa
└── id_rsa.pub
First, name that key pair – adding a descriptive name will help you remember which key is used for which user/remote
# change to your ~/.ssh directory
$ cd ~/.ssh
# rename the private key
$ mv id_rsa github-mainuser
# rename the public key
$ mv id_rsa.pub github-mainuser.pub
Next, let's generate a new key pair – here I'll name the new key github-otheruser
$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/github-otheruser
Now, when we look at tree ~/.ssh we see
$ tree ~/.ssh
/Users/you/.ssh
├── known_hosts
├── github-mainuser
├── github-mainuser.pub
├── github-otheruser
└── github-otheruser.pub
Next, we need to setup a ~/.ssh/config file that will define our key configurations. We'll create it with the proper owner-read/write-only permissions
$ (umask 077; touch ~/.ssh/config)
Open that with your favourite editor, and add the following contents
Host github.com
User git
IdentityFile ~/.ssh/github-mainuser
Host github.com-otheruser
HostName github.com
User git
IdentityFile ~/.ssh/github-otheruser
Presumably, you'll have some existing repos associated with your primary github identity. For that reason, the "default" github.com Host is setup to use your mainuser key. If you don't want to favour one account over another, I'll show you how to update existing repos on your system to use an updated ssh configuration.
Add your new SSH key to github
Head over to github.com/settings/keys to add your new public key
You can get the public key contents using: copy/paste it to github
$ cat ~/.ssh/github-otheruser.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDBVvWNQ2nO5...
Now your new user identity is all setup – below we'll show you how to use it.
Getting stuff done: cloning a repo
So how does this come together to work with git and github? Well because you can't have a chicken without and egg, we'll look at cloning an existing repo. This situation might apply to you if you have a new github account for your workplace and you were added to a company project.
Let's say github.com/someorg/somerepo already exists and you were added to it – cloning is as easy as
$ git clone github.com-otheruser:someorg/somerepo.git
That bolded portion must match the Host name we setup in your ~/.ssh/config file. That correctly connects git to the corresponding IdentityFile and properly authenticates you with github
Getting stuff done: creating a new repo
Well because you can't have a chicken without and egg, we'll look at publishing a new repo on your secondary account. This situation applies to users that are create new content using their secondary github account.
Let's assume you've already done a little work locally and you're now ready to push to github. You can follow along with me if you'd like
$ cd ~
$ mkdir somerepo
$ cd somerepo
$ git init
Now configure this repo to use your identity
$ git config user.name "Mister Manager"
$ git config user.email "someuser#some.org"
Now make your first commit
$ echo "hello world" > readme
$ git add .
$ git commit -m "first commit"
Check the commit to see your new identity was used using git log
$ git log --pretty="%H %an <%ae>"
f397a7cfbf55d44ffdf87aa24974f0a5001e1921 Mister Manager <someuser#some.org>
Alright, time to push to github! Since github doesn't know about our new repo yet, first go to github.com/new and create your new repo – name it somerepo
Now, to configure your repo to "talk" to github using the correct identity/credentials, we have add a remote. Assuming your github username for your new account is someuser ...
$ git remote add origin github.com-otheruser:someuser/somerepo.git
That bolded portion is absolutely critical and it must match the Host that we defined in your ~/.ssh/config file
Lastly, push the repo
$ git push origin master
Update an existing repo to use a new SSH configuration
Say you already have some repo cloned, but now you want to use a new SSH configuration. In the example above, we kept your existing repos in tact by assigning your previous id_rsa/id_rsa.pub key pair to Host github.com in your SSH config file. There's nothing wrong with this, but I have at least 5 github configurations now and I don't like thinking of one of them as the "default" configuration – I'd rather be explicit about each one.
Before we had this
Host github.com
User git
IdentityFile ~/.ssh/github-mainuser
Host github.com-otheruser
HostName github.com
User git
IdentityFile ~/.ssh/github-otheruser
So we will now update that to this (changes in bold)
Host github.com-mainuser
HostName github.com
User git
IdentityFile ~/.ssh/github-mainuser
Host github.com-otheruser
HostName github.com
User git
IdentityFile ~/.ssh/github-otheruser
But now any existing repo with a github.com remote will not work with this identity file. But don't worry, it's a simple fix.
To update any existing repo to use your new SSH configuration, update the repo's remote origin field using set-url -
$ cd existingrepo
$ git remote set-url origin github.com-mainuser:someuser/existingrepo.git
That's it. Now you can push/pull to your heart's content
SSH key file permissions
If you're running into trouble with your public keys not working correctly, SSH is quite strict on the file permissions allowed on your ~/.ssh directory and corresponding key files
As a rule of thumb, any directories should be 700 and any files should be 600 - this means they are owner-read/write-only – no other group/user can read/write them
$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/config
$ chmod 600 ~/.ssh/github-mainuser
$ chmod 600 ~/.ssh/github-mainuser.pub
$ chmod 600 ~/.ssh/github-otheruser
$ chmod 600 ~/.ssh/github-otheruser.pub
How I manage my SSH keys
I manage separate SSH keys for every host I connect to, such that if any one key is ever compromised, I don't have to update keys on every other place I've used that key. This is like when you get that notification from Adobe that 150 million of their users' information was stolen – now you have to cancel that credit card and update every service that depends on it – what a nuisance.
Here's what my ~/.ssh directory looks like: I have one .pem key for each user, in a folder for each domain I connect to. I use .pem keys to so I only need one file per key.
$ tree ~/.ssh
/Users/myuser/.ssh
├── another.site
│   ├── myuser.pem
├── config
├── github.com
│   ├── myuser.pem
│   ├── someusername.pem
├── known_hosts
├── somedomain.com
│   ├── someuser.pem
└── someotherdomain.org
   └── root.pem
And here's my corresponding /.ssh/config file – obviously the github stuff is relevant to answering this question about github, but this answer aims to equip you with the knowledge to manage your ssh identities on any number of services/machines.
Host another.site
User muyuser
IdentityFile ~/.ssh/another.site/muyuser.pem
Host github.com-myuser
HostName github.com
User git
IdentityFile ~/.ssh/github.com/myuser.pem
Host github.com-someuser
HostName github.com
User git
IdentityFile ~/.ssh/github.com/someusername.pem
Host somedomain.com
HostName 162.10.20.30
User someuser
IdentityFile ~/.ssh/somedomain.com/someuser.pem
Host someotherdomain.org
User someuser
IdentityFile ~/.ssh/someotherdomain.org/root.pem
Getting your SSH public key from a PEM key
Above you noticed that I only have one file for each key. When I need to provide a public key, I simply generate it as needed.
So when github asks for your ssh public key, run this command to output the public key to stdout – copy/paste where needed
$ ssh-keygen -y -f someuser.pem
ssh-rsa AAAAB3NzaC1yc2EAAAA...
Note, this is also the same process I use for adding my key to any remote machine. The ssh-rsa AAAA... value is copied to the remote's ~/.ssh/authorized_keys file
Converting your id_rsa/id_rsa.pub key pairs to PEM format
So you want to tame you key files and cut down on some file system cruft? Converting your key pair to a single PEM is easy
$ cd ~/.ssh
$ openssl rsa -in id_rsa -outform pem > id_rsa.pem
Or, following along with our examples above, we renamed id_rsa -> github-mainuser and id_rsa.pub -> github-mainuser.pub – so
$ cd ~/.ssh
$ openssl rsa -in github-mainuser -outform pem > github-mainuser.pem
Now just to make sure that we've converted this correct, you will want to verify that the generated public key matches your old public key
# display the public key
$ cat github-mainuser.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAA ... R++Nu+wDj7tCQ==
# generate public key from your new PEM
$ ssh-keygen -y -f someuser.pem
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAA ... R++Nu+wDj7tCQ==
Now that you have your github-mainuser.pem file, you can safely delete your old github-mainuser and github-mainuser.pub files – only the PEM file is necessary; just generate the public key whenever you need it ^_^
Creating PEM keys from scratch
You don't need to create the private/public key pair and then convert to a single PEM key. You can create the PEM key directly.
Let's create a newuser.pem
$ openssl genrsa -out ~/.ssh/newuser.pem 4096
Getting the SSH public key is the same
$ ssh-keygen -y -f ~/.ssh/newuser.pem
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACA ... FUNZvoKPRQ==
By creating different host aliases to github.com in your ~/.ssh/config,
and giving each host alias its own ssh key, you can easily use multiple
github accounts without confusion.
That’s because github.com distinguishes not by user, which is always just git,
but by the ssh key you used to connect.
Just configure your remote origins using your own host aliases.”
The above summary is courtesy of comments on the blog post below.
I've found this explanation the clearest. And it works for me, at least as of April 2012.
http://net.tutsplus.com/tutorials/tools-and-tips/how-to-work-with-github-and-multiple-accounts/
The details at http://net.tutsplus.com/tutorials/tools-and-tips/how-to-work-with-github-and-multiple-accounts/ linked to by mishaba work very well for me.
From that page:
$ touch ~/.ssh/config
Then edit that file to be something like this (one entry per account):
#Default GitHub
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa
Host github-COMPANY
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa_COMPANY
In my case, I have my work account in Git-Lab and my personal account in GitHub.
My Git-Lab account was configured globally to be accessed from all directories in my laptop like this:
git config --global user.name WORK_USERNAME
git config --global user.email WORK_USERNAME#example.com
So if you're looking for a solution without using SSL, you can do it by using git init in an empty folder, and then insert your personal account credentials in that folder:
git config user.name PERSONAL_USERNAME
git config user.email PERSONAL_USERNAME#example.com
Notice here --global is not set, because you only want your personal git to be accessed from there and not everywhere, so whatever inside that folder will have its git credentials connected to your personal account and outside of it will be connected to your work account.
After that, you can clone your repository like so git clone your_repo_link.git. Then a new window will popup and ask you to login with your github account.
To verify your process try git config --list inside the folder that you created, you should see both work and personal usernames and emails with their directories.
Outside that folder, if you run git config --list you should see your work username and email only.
That's it, hope this helps someone without configuring SSL.
Go to ~/.ssh
Create a file named config(have no extension )
Open config file & add below codes. (change according to your account)
Account 1
# account_1
Host gitlab.com-account_1
HostName gitlab.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_account_1
Account 2
# Account2
Host gitlab.com-Account2
HostName gitlab.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_Account2
Account 3
# Account_3
Host github.com-Account3
HostName github.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_Account_3
Add remote url as follows
Account 1
git remote add origin git#gitlab.com-account_1:group_name/repo_name.git
Account 2
git remote add origin git#gitlab.com-Account2:group_name/repo_name.git
Account 3
git remote add origin github.com-Account3:github_username/repo_name.git
Make sure that IdentityFile names are same as you created during ssh key generation.
This answer is for beginners (none-git gurus). I recently had this problem and maybe its just me but most of the answers seemed to require rather advance understanding of git. After reading several stack overflow answers including this thread, here are the steps I needed to take in order to easily switch between GitHub accounts (e.g. assume two GitHub accounts, github.com/personal and gitHub.com/work):
Check for existing ssh keys: Open Terminal and run this command to see/list existing ssh keys ls -al ~/.ssh files with extension .pub are your ssh keys so you should have two for the personal and work accounts. If there is only one or none, its time to generate other wise skip this.- Generating ssh key: login to github (either the personal or work acc.), navigate to Settings and copy the associated email.now go back to Terminal and run ssh-keygen -t rsa -C "the copied email", you'll see:Generating public/private rsa key pair.
Enter file in which to save the key (/.../.ssh/id_rsa): id_rsa is the default name for the soon to be generated ssh key so copy the path and rename the default, e.g. /.../.ssh/id_rsa_work if generating for work account. provide a password or just enter to ignore and, you'll read something like The key's randomart image is: and the image. done.Repeat this step once more for your second github account. Make sure you use the right email address and a different ssh key name (e.g. id_rsa_personal) to avoid overwriting. At this stage, you should see two ssh keys when running ls -al ~/.ssh again.
Associate ssh key with gitHub account: Next step is to copy one of the ssh keys, run this but replacing your own ssh key name: pbcopy < ~/.ssh/id_rsa_work.pub, replace id_rsa_work.pub with what you called yours.Now that our ssh key is copied to clipboard, go back to github account [Make sure you're logged in to work account if the ssh key you copied is id_rsa_work] and navigate toSettings - SSH and GPG Keys and click on New SSH key button (not New GPG key btw :D) give some title for this key, paste the key and click on Add SSH key. You've now either successfully added the ssh key or noticed it has been there all along which is fine (or you got an error because you selected New GPG key instead of New SSH key :D).
Associate ssh key with gitHub account: Repeat the above step for your second account.
Edit the global git configuration: Last step is to make sure the global configuration file is aware of all github accounts (so to say). Run git config --global --edit to edit this global file, if this opens vim and you don't know how to use it, press i to enter Insert mode, edit the file as below, and press esc followed by :wq to exit insert mode:
[inside this square brackets give a name to the followed acc.]
name = github_username
email = github_emailaddress
[any other name]
name = github_username
email = github_email
[credential]
helper = osxkeychain
useHttpPath = true
Done!, now when trying to push or pull from a repo, you'll be asked which GitHub account should be linked with this repo and its asked only once, the local configuration will remember this link and not the global configuration so you can work on different repos that are linked with different accounts without having to edit global configuration each time.
I use shell scripts to switch me to whatever account I want to be "active". Essentially you start from a fresh start, get one account configured properly and working, then move the these files to a name with the proper prefix. From then on you can use the command "github", or "gitxyz" to switch:
# my github script
cd ~/.ssh
if [ -f git_dhoerl -a -f git_dhoerl.pub -a -f config_dhoerl ]
then
;
else
echo "Error: missing new files"
exit 1
fi
# Save a copy in /tmp, just in case
cp id_rsa /tmp
cp id_rsa.pub /tmp
cp config /tmp
echo "Saved old files in /tmp, just in case"
rm id_rsa
rm id_rsa.pub
rm config
echo "Removed current links/files"
ln git_dhoerl id_rsa
ln git_dhoerl.pub id_rsa.pub
ln config_dhoerl config
git config --global user.email "dhoerl#<company>.com"
git config --global github.user "dhoerl"
git config --global github.token "whatever_it_is"
ssh-add -D
I've had great luck with this. I also created a run script in Xcode (for you Mac users) so it would not build my project unless I had the proper setting (since its using git):
Run Script placed after Dependencies (using /bin/ksh as the shell):
if [ "$(git config --global --get user.email)" != "dhoerl#<company>.com" ]
then
exit 1
fi
EDIT: added tests for new files existence and copying old files to /tmp to address comment by #naomik below.
just figured this out for Windows, using credentials for each repo:
cd c:\User1\SomeRepo
git config --local credential.https://github.com.user1 user1
git config --local credential.useHttpPath true
git config --local credential.helper manager
git remote set-url origin https://USERNAME#github.com/USERNAME/PROJECTNAME.git
The format of credential.https://github.com. tells the credential helper the URL for the credential.
The 'useHttpPath' tells the credential manager to use the path for the credential. If useHttpPath is omitted then the credential manager will store one credential for https://github.com. If it is included then the credential manager will store multiple credentials, which is what I really wanted.
Simpler and Easy fix to avoid confusion..
For Windows users to use multiple or different git accounts for different projects.
Following steps:
Go Control Panel and Search for Credential Manager.
Then Go to Credential Manager -> Windows Credentials
Now remove the git:https//github.com node under Generic Credentials Heading
This will remove the current credentials. Now you can add any project through git pull it will ask for username and password.
When you face any issue with other account do the same process.
There are plenty of answers explaining how to accomplish what you have asked but I'd like to offer a different perspective.
I think there's something to be said for keeping work and personal stuff separate from each other. It's pretty trivial to switch between user accounts on all operating systems so you could just create separate OS accounts for work and play and have different github accounts logged in in each.
Maintaining a clear separation of concerns can also...
prevent your work files becoming co-mingled with your personal files
reduce the risk of carrying out an action on the wrong resource
potentially confine security breaches to one set of accounts / credentials
In case you don't want to mess with the ~/.ssh/config file mentioned here, you could instead run git config core.sshCommand "ssh -i ~/.ssh/custom_id_rsa" in the repo where you want to commit from a different account.
The rest of the setup is the same:
Create a new SSH key for the second account with ssh-keygen -t rsa -f ~/.ssh -f ~/.ssh/custom_id_rsa
Sign in to github with your other account, go to https://github.com/settings/keys , and paste the contents of ~/.ssh/custom_id_rsa.pub
Make sure you're using SSH instead of HTTPS as remote url: git remote set-url origin git#github.com:upstream_project_teamname/upstream_project.git
I see a lot of possible workarounds here. As a quick fix, #Greg's one seems relatively easy. However, it's best to set up separate ssh keys for all different accounts in the long run. This video demonstrates it briefly. Here are the steps mentioned in that video blog.
Step 1 - Create a New SSH Key for a new account and save it in a separate file (e.g. ~/.ssh/id_rsa_new_account_name), not in the original file i.e. ~/.ssh/id_rsa
ssh-keygen -t rsa -C "your-email-address"
Step 2 - Attach the New Key
Next, login to your second GitHub account
Browse to Account Overview, and attach the new key ~/.ssh/id_rsa_new_account_name.pub within the SSH Public Keys section.
In the Terminal, tell SSH to add the new identity by typing ssh-add ~/.ssh/id_rsa_new_account_name. If successful, you'll see a response of Identity Added.
Step 3 - Create a Config File
touch ~/.ssh/config
And save the following contents into the file
#Default GitHub
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa
#New Account
Host github-new-account-name
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa_new_account_name
Step 4 - Try it Out
Now every time you want to use the new account and the associated repo, type this for the respective repo
git remote add origin git#github-new-account-name:your-domain.com/your-repo.git
Beside of creating multiple SSH Keys for multiple accounts you can also consider to add collaborators on each project using the same account emails and store the password permanently.
#this store the password permanently
$ git config --global credential.helper wincred
I have setup multiple accounts with different emails then put the same user and email on each account as one of the collaborators. By this way I can access to all account without adding SSH Key, or switching to another username, and email for the authentication.
Got my private repo working using SSH key pairs. This was tested on git for Windows.
Source: https://docs.github.com/en/free-pro-team#latest/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
A. Generate public and private key pairs
Start git bash
Run ssh-keygen -t ed25519 -C "your_email#example.com"
When you're prompted to "Enter a file in which to save the key," press Enter to accept the default.
Press enter for a blank passphrase.
Start the ssh agent: eval $(ssh-agent)
Add private key to ssh agent and store the passphrase: ssh-add ~/.ssh/id_ed25519
B. Add SSH keys to GitHub account
Copy the public key to the clipboard: clip < ~/.ssh/id_ed25519.pub
On GitHub, go to Profile -> Settings -> SSH Keys -> New SSH Key
Give a title. E.g. "Windows on MacBook Pro"
Paste the key and hit "Add SSH Key".
C. Test SSH connection
Enter: ssh -T git#github.com
Hit "yes" for any warning message.
It should show: "Hi username!..." indicating a successful test.
D. Setup local repository to use SSH keys
Change email and user name:
git config user.email your_email#example.com
git config user.name github_username
Update remote links to use git. First list remote URI's:
git remote -v
git remote set-url origin git#github.com:github_username/your-repo-name.git
E. Test
git remote show origin
Simple approach without dealing with SSH keypairs
Just change the user.name & user.email locally in the repo. to the ones you want to push to that repo with.
Example: I have a work account in gitlab having a project. After cloning this project/repo, in the terminal, I type out:
git config user.name "my-work-username"
git config user.email "my-work-email-id"
Now, say I've another personal project/repo in Gitlab that I want to associate with my personal account. Just like above, after cloning, I type:
git config user.name "my-personal-username"
git config user.email "my-personal-email-id"
Hope this helps. Upvote if it worked for you! :)
The easiest and straightforward approach (IMHO) - no config files not too much hassle
Just create another ssh key.
Let's say you have a new work GitHub account, just create a new key for it:
ssh-keygen -t rsa -C "email#work_mail.com" -f "id_rsa_work_user1"`
You need to run the above only once.
Now you should have the old one and the new one, to see them, run:
ls -al ~/.ssh
From now on, every time you want to switch between the two, just run:
ssh-add -D
ssh-add ~/.ssh/id_rsa_work_user1 #make to use this without the suffix .pub
In order the switch to the old one, run again:
ssh-add -D
ssh-add ~/.ssh/<previous id_rsa>
I found this gem to be very useful: sshwitch
https://github.com/agush22/sshwitch
http://rubygems.org/gems/sshwitch
It helps to switch out ssh keys. Remember to back up everything first!
Also to make sure that commits have the correct email address associated with them, I made sure that the ~/.gitconfig file had the proper email address.
another easier way is using multiple desktop apps, like what i am doing, using account A on Github desktop, while using account B on Github Kraken
If you happen to have WSL installed you can have two seperate git accounts - one on WSL and one in windows.
IntelliJ Idea has built-in support of that
https://www.jetbrains.com/help/idea/github.html#da8d32ae
Personal Directory .gitconfig using a personal access token
If you do not want to modify your host file, use SSH keys, or setup a .gitconfig for each repo, then you may use a personal .gitconfig that you basically include from the root level config. Given an OSX directory structure like
# root level git config
~/.gitconfig
# your personal repos under some folder like
../personal/
../home/
~/Dropbox/
Add a .gitconfig in your personal folder, such as ~/Dropbox/.gitconfig
[user]
email = first.last#home.com
name = First Last
[credential]
username = PersonalGithubUsername
helper = osxkeychain
In your root level .gitconfig add an includeIf section to source your personal config whenever you are in your personal directory. Any settings there will override the root config as long as the includeIf comes after the settings you want to override.
[user]
email = first.last#work.com
name = "First Last"
[credential]
helper = osxkeychain
[includeIf "gitdir:~/Dropbox/**"]
path = ~/Dropbox/.gitconfig
Try pushing to your personal repo or pulling from your private repo
git push
# prompts for password
When prompted enter either your personal password or, better yet, your personal access token that you have created in your account developer settings. Enter that token as your password.
Assuming you are already using git-credential-osxkeychain, your personal credentials should be stored in your keychain, so two github entries will show up, but with different accounts.
Option 0: you dont want to mess around with OS settings.. you just want to commit to a different github account with a different public key for one repo.
solution:
create the new key: ssh-keygen -t rsa -b 4096 -f ~/.ssh/alt_rsa
add the key to the keyset: ssh-add -K ~/.ssh/alt_rsa
copy and add the pub key to the github account: (see github instructions)
test the key with github: ssh -i ~/.ssh/alt_rsa T git#github.com
clone the repo using the git protocol (not HTTP): git clone git#github:myaccount...
in the cloned repo:
git config core.sshCommand "ssh -i ~/.ssh/alt_rsa -F /dev/null"
git config user.name [myaccount]
git config user.email [myaccount email]
now you should be able to git push correctly without interferring with your everyday git account
Manage multiple GitHub accounts on one Windows machine (HTTPS)
Let's say you previously use git on your machine and configure git global config file. To check it open the terminal and :
git config --global -e
It opens your editor, and you may see something like this:
[user]
email = yourEmail#gmail.com
name = Your_Name
...
And this is great because you can push your code to GitHub account without entering credentials every time.
But what if it needs to push to repo from another account? In this case, git will reject with 403 err, and you must change your global git credentials. To make this comfortable lat set storing a repo name in a credential manager:
git config --global credential.github.com.useHttpPath true
to check it open config one more time
git config --global -e
you will see new config lines
[credential]
useHttpPath = true
...
The is it. Now when you first time push to any account you will see a pop-up
Screenshot_1
Enter specific for this repo account credentials, and this will "bind" this account for the repo. And so in your machine, you can specify as many accounts/repos as you want.
For a more expanded explanation you can see this cool video that I found on youtube:
https://youtu.be/2MGGJtTH0bQ
Use docker!
This may not be suitable for everyone but this is what I ended up doing for myself.
All my developments are now done inside docker containers. I am a vim user but I know VSCode has plugin to work with docker containers.
So since I have different docker containers for different projects/languages, I can use different git configurations inside each. Problem solved.
just add this line in your fav editor and you are done for life
git remote set-url origin https://user-token-of-particular-user#github.com/profile-name/repo-name
Unlike other answers, where you need to follow few steps to use two different github account from same machine, for me it worked in two steps.
You just need to :
1) generate SSH public and private key pair for each of your account under ~/.ssh location with different names and
2) add the generated public keys to the respective account under Settings >> SSH and GPG keys >> New SSH Key.
To generate the SSH public and private key pairs use following command:
cd ~/.ssh
ssh-keygen -t rsa -C "email#work.com" -f "id_rsa_WORK"
ssh-keygen -t rsa -C "email#gmail.com" -f "id_rsa_PERSONAL"
As a result of above commands, id_rsa_WORK and id_rsa_WORK.pub files will be created for your work account (ex - git.work.com) and id_rsa_PERSONAL and id_rsa_PERSONAL.pub will be created for your personal account (ex - github.com).
Once created, copy the content from each public (*.pub) file and do Step 2 for the each account.
PS : Its not necessary to make an host entry for each git account under ~/.ssh/config file as mentioned in other answers, if hostname of your two accounts are different.
You should and must not push to the project with some common credentials. Once starting on a new machine use the following steps to setup and use correctly your gitlab credentials:
create the pubic / private ssh keys on the machine
copy paste the public key to the gitlab/github ui interface ( anyone hinting how-to do via the cmd line gets a free beer ... )
make sure you clone the repo via the git and not http url
set the git alias to avoid constant typing of the same prefix to the git command
during git commit ALWAYS use the author and e-mail flags
use git as normal you would do it
All this as follows:
# create the public / private key credentials on that specific machine
ssh-keygen -t rsa -b 4096 -C "<<you>>#org.net" -f ~/.ssh/id_rsa.<<you>>.`hostname -s`
# setup your public key in the gitlab ui
cat ~/.ssh/id_rsa.<<you>>.`hostname -s`
# make sure you clone the repo via the git and not http url
git clone git#git.in.org.net:org/some-repo.git
# set the git alias to avoid constant typing of the repeating prefix to the git cmd
alias git='GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa.<<you>>.`hostname -s`" git'
# during git commit ALWAYS use the author and e-mail flags
git add --all ; git commit -nm "$git_msg" --author "YourFirstName YourLastName <you#phz.fi>"
# use git as normal
git fetch --all; git pull --all
Navigate to the directory in which you want to push your changes to a different GitHub account.
Create a new SSH key in your terminal/command line.
ssh-keygen -t rsa -C “your-email-address”
The following will then show:
Generating public/private rsa key pair.
Enter file in which to save the key (/home/your_username/.ssh/id_rsa):
Copy and paste the path followed by an identifiable name for the file:
/home/your_username/.ssh/id_rsa_personal
4) It will then ask you for the following:
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
5) You can now type in the following command to see all the SSH keys you have on your local machine:
ls -al ~/.ssh
You should be able to see your new SSH key file. As you can see in my one I have both id_rsa_test and id_rsa_personal.pub.
drwx------ 2 gmadmin gmadmin 4096 Nov 16 22:20 .
drwxr-xr-x 42 gmadmin gmadmin 4096 Nov 16 21:03 ..
-rw------- 1 gmadmin gmadmin 1766 Nov 16 22:20 id_rsa_personal
-rw-r--r-- 1 gmadmin gmadmin 414 Nov 16 22:20 id_rsa_personal.pub
-rw-r--r-- 1 gmadmin gmadmin 444 Nov 6 11:32 known_hosts
6) Next you need to copy the SSH key which is stored in id_rsa_personal.pub file. You can open this in text editor of your choice. I am currently using atom so I opened the file using the following command:
atom ~/.ssh/id_rsa_personal.pub
You will then get something similar to this:
ssh-rsa AAB3HKJLKC1yc2EAAAADAQABAAABAQCgU5+ELtwsKkmcoeF3hNd7d6CjW+dWut83R/Dc01E/YzLc5ZFri18doOwuQoeTPpmIRVDGuQQsZshjDrTkFy8rwKWMlXl7va5olnGICcpg4qydEtsW+MELDmayW1HHsi2xHMMGHlNv
7) Copy this and navigate to your GitHub account → Settings → SSH and GPG keys
8) Click on New SSH key. Copy the key, give it a title and add it.
9) Add key from terminal
ssh-add ~/.ssh/id_rsa_personal
Enter passphrase for /home/your_username/.ssh/id_rsa_personal:
10) Configure user and password.
git config --global user.name "gitusername"
git config --global user.email "gitemail"
11) We are ready to commit and push now.
git init
git add .
git commit
git push

How can gitlab-CI install private python packages from a gitlab dependency that also refers to gitlab repositories

This question is about how to manage access for private gitlab python packages with nested dependencies on other private gitlab packages. This assumes all the access is via direct git repository patterns and not a private package repository.
package-a is in a private gitlab repository and it depends on package-b, which depends on package-c and they are also in a private gitlab repository.
package-a has a pyproject.toml like this:
[tool.poetry]
name = "package-a"
repository = "https://gitlab.com/org/package_a.git"
[tool.poetry.dependencies]
python = "^3.6"
package-b = {git = "ssh://git#gitlab.com/org/package_b.git", tag = "0.1.0"}
package-b has a pyproject.toml like this:
[tool.poetry]
name = "package-b"
repository = "https://gitlab.com/org/package_b.git"
[tool.poetry.dependencies]
python = "^3.6"
package-c = {git = "ssh://git#gitlab.com/org/package_c.git", tag = "0.1.0"}
Any user with the right org membership on gitlab and an ssh-key can use poetry to install package-a and it's dependency on package-b and then it's dependency on package-c, all into a python venv on a development laptop. The ssh protocol access also works for docker builds (with experimental features for ssh mounts).
However, the same projects with private dependencies are not installed in gitlab-CI runners, because they lack ssh access. (Is there any secure way to enable that?)
Assuming the gitlab-CI runners must use an access token to clone the private gitlab repositories, a sed script is applied to the pyproject.toml file for project-a, so the gitlab-CI runner can clone package-b to discover that it depends on package-c; the sed script changes the ssh to https access for project-b by editing the project-a dependency spec in pyproject.toml, i.e.
sed -i -e 's#ssh://git#gitlab.com/org#https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com/org#g' pyproject.toml
The CI_JOB_TOKEN is an environment variable provided by the gitlab-CI runner. It is securely managed by the gitlab-CI runner. So, gitlab-CI runner can now clone the project-b repository somewhere. The same sed trick might work if it could be applied to this project-b repository somewhere, but this is now in the hands of poetry and it cannot be touched. So project-b has a git+ssh dependency on project-c and the gitlab-CI runner fails to install project-c because it has no git+ssh credentials to clone it.
Thus, the private package dependency chain works for development and docker build on a laptop with git+ssh access, but it's all broken on gitlab-CI. What's the better practice to manage this private package access across all these build environments?
These snippets are based on:
https://docs.gitlab.com/ee/ci/ssh_keys/
https://gitlab.com/gitlab-examples/ssh-private-key
https://gitlab.com/gitlab-examples/ssh-private-key/issues/7
https://gitlab.com/gitlab-examples/ssh-private-key/issues/1
ssh-keygen -o -t rsa -b 4096 -C "git#gitlab.com"
# output to something like ~/.ssh/gitlab_ci_rsa
# do not add any passphrase
# once created, copy the private key to the clipboard, e.g.
cat ~/.ssh/gitlab_ci_rsa | base64 -w0 > tmp.txt
xclip -sel clip < tmp.txt
The public key is used as a private deploy key,
which is enabled from a project settings page, e.g.
https://gitlab.com/org/project-a/settings/repository
https://gitlab.com/org/project-b/settings/repository
https://gitlab.com/org/project-c/settings/repository
The private key is pasted into the gitlab-CI variable SSH_PRIVATE_KEY
and gitlab should be able to mask it (when it is base64 encoded). Then the .gitlab-ci.yml file can add this private key to the ssh-agent using:
before_script:
- apt-get update -y -qq && apt-get install -y -qq git make openssh-client
- eval $(ssh-agent -s)
## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 --decode)
## Create the SSH directory and give it the right permissions
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
The gitlab documentation does not use base64 encoding, but it's essential to copying all of the private key into the variable and that avoids a prompt for a passphrase from ssh-add.

How to deploy private python pip dependency with Amazon AWS Elastic Beanstalk?

When I tried to set one of the services into AWS Elastic Beanstalk, the problem appeared. One of our python pip dependencies set up on private repository was causing errors during deployment process, since it was not accessible for pip process. Bellow I present description how we solved this issue.
It is worth to mention that there are other solutions that bind SSH keys used during deployment, to application project git repository. I find them a bit dirty, so I would like to share this one that allows to keep SSH keys separated in S3 bucket, separated from application git repository.
Works on: "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4" for dependency from private bitbucket repository. it may be modified for github, etc.
Add you private dependency to pip requirements.txt:
-e git+git#bitbucket.org:account_name/dependency-name.git#egg=dependency-name
Generate public & private SSH keys (HOWTO can be found elsewhere) so you have id_rsa (private) and id_rsa.pub (public) key files.
On bitbucket.org project tab, find settings and go under "Deployment keys". Use form to set your public SSH key there.
Upload both generated keys (just private should be enough) on S3 bucket using amazon AWS console:
bucket-with-keys/bitbucket/:
- id_rsa
- id_rsa.pub
Where bucket-with-keys/bitbucket - is [BUCKET_NAME]/[PATH].
Add packages file to your project: project_name/.ebextensions/packages.config
packages:
yum:
git: []
Add configuration file to your project: project_name/.ebextensions/03-pip-install-from-bitbucket.config:
files:
"/root/.ssh/config":
owner: root
group: root
mode: "000600"
content: |
Host *
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
"/root/.ssh/known_hosts":
owner: root
group: root
mode: "000644"
content: |
#
# paste output of `ssh-keyscan -H github.com` here
#
commands:
01-command:
command: aws s3 cp --recursive s3://bucket-with-keys/bitbucket/ /root/.ssh
02-command:
command: chmod 600 /root/.ssh/id_rsa
If you modify this file, keep in mind that:
commands are being executed in alphabetical order so "01-" starts before "02-" even if you would switch its order.
file name also follows that rule.
Go to AWS console IAM (Identity & Access Management) "policies" tab and find AmazonS3FullAccess on a list and attache it for aws-elasticbeanstalk-ec2-role.
If you cannot find policy you can create one like below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
Look for "Create and Attach Your First Customer Managed Policy" Tutorial if needed. This policy is quite open and it is advised to create one that would be more narrow...
aws-elasticbeanstalk-ec2-role is role created by default by Elastic Beanstalk but you can use your own as long as it is set with CLI tool eb config under: "IamInstanceProfile: aws-elasticbeanstalk-ec2-role"
Now you can create your environmant with eb CLI tool:
eb create project_name-prod-env \
--instance_type t2.micro \
--tier webserver \
--region eu-west-1 \
--cname project_name-prod-env \
--keyname identifier-of-ssh-key-accessed-from-console-here \
--platform "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4"
Should work now!
If sth goes wrong you may debug if SSH files got to its place:
eb ssh
sudo su
ls -la /root/.ssh/
And check logs on AWS console or directly on instance:
eb ssh
sudo su
less /var/log/eb-activity.log
Try manually execute commands from project_name/.ebextensions/03-pip-install-from-bitbucket.config as a root user bu in the way they appear in log file, use switches to get more verbose output.
I cannot comment so I'm answering the question. smentek's answer is very detailed and solves the issue. The only missing part is that you need to add the git package to the config:
packages:
yum:
git-all: ""
files:
"/root/.ssh/config":
# ...
Git isn't installed on Amazon Linux by default.

Python Fabric and Amazon EC2 : Sudo password asked

You know that on EC2, there is no password associated with "ubuntu" user. With the following lines, if I try to run :
fab development install_dir
I get :
[ec2-46-51-132-252.eu-west-1.compute.amazonaws.com] sudo: chown -R webadmin:webadmin /var/www
[ec2-46-51-132-252.eu-west-1.compute.amazonaws.com] Login password:
I tried to add shell=False to sudo method (according to Can I prevent fabric from prompting me for a sudo password?), but it doesn't change anything
Any idea ? Thanks a lot !
def development():
env.envname = 'development'
env.user = 'ubuntu'
env.group = 'ubuntu'
env.chuser = 'webadmin'
env.chgroup = 'webadmin'
env.hosts = ['ec2-***.eu-west-1.compute.amazonaws.com']
env.envname_abriev = 'dev'
env.key_filename = '/home/xx/.ssh/xx.pem'
env.postgresql_version = '9.0'
def install_dir():
if not exists('/var/www'):
sudo('mkdir /var/www')
sudo('chown -R %s:%s /var/www' % (env.chuser, env.chgroup))
Download (or create) a keypair file from aws as shown below
Create a file called fabfile.py
and set its contents as follows:
from fabric.context_managers import cd
from fabric.operations import sudo
from fabric.api import run, env
import os
HOME = os.getenv('HOME')
env.user = 'ubuntu'
env.hosts = ['PUBLICDNS.ap-southeast-1.compute.amazonaws.com','ANOTHERSERVER.compute.amazonaws.com'] #can add multiple instances
env.key_filename = [
'%s/<your-keypair-file>.pem'%HOME
] #assuming keypair file is located in HOME
#example code we want to run on remote machine
def update():
with cd('/var/www'):
sudo('svn update')
with cd ('/var/www/cache'):
run('rm -rf *')
sudo('service lighttpd restart')
To run the file, type fab update in the terminal.
You need to specify keypair file name associated with your EC2 instance while running fab command.
Usage: fab [options] <command>[:arg1,arg2=val2,host=foo,hosts='h1;h2',...] ...
Options:
-R ROLES, --roles=ROLES
comma-separated list of roles to operate on
-i KEY_FILENAME path to SSH private key file. May be repeated.

Categories

Resources