Develop Reference

GitLab runner, error on push whereas same thing works on git bash or python from a console

Context
We're trying to do a GitLab runner job that, on a certain tag, modifies a version header file and add a release branch/tag to this changeset.
The GitLab runner server is on my machine, launched as a service by my user (that is properly registered to our GitLab server).
The GitLab runner job basically launches a python script that uses gitpython to du the job, there are just a few changes in runner yml file (added before_script part to be able to have upload permission, got it from there: https://stackoverflow.com/a/55344804/11159476), here is full .gitlab-ci.yml file:
variables:
GIT_SUBMODULE_STRATEGY: recursive
stages: [ build, publish, release ]
release_tag:
stage: build
before_script:
- git config --global user.name ${GITLAB_USER_NAME}
- git config --global user.email ${GITLAB_USER_EMAIL}
script:
- python .\scripts\release_gitlab_runner.py
only:
# Trigger on specific regex...
- /^Src_V[0-9]+\.[0-9]+\.[0-9]+$/
except:
# .. only for tags then except branches, see doc (https://docs.gitlab.com/ee/ci/yaml/#regular-expressions): "Only the tag or branch name can be matched by a regular expression."
- branches
Also added trick in the python URL when pushing (push with user:personal_access_token#repo_URL instead of default runner URL, got it from same answer as above, and token has been generated from company gitlab => user "Settings" => "Access Tokens" => "Add a personal access token" with all rights and never expiring), and here is, not the actual scripts\release_gitlab_runner.py python script but one simplified to have a git flow as much standard as possible for what we want (fetch all, create local branch with random name so that it does not exist, modify a file, stage, commit and finally push):
# -*-coding:utf-8 -*
import uuid
import git
import sys
import os
# Since we are in <git root path>/scripts folder, git root path is this file's path parent path
GIT_ROOT_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
try:
# Get user login and URL from GITLAB_USER_LOGIN and CI_REPOSITORY_URL gitlab environment variables
gitlabUserLogin = os.environ["GITLAB_USER_LOGIN"]
gitlabFullURL = os.environ["CI_REPOSITORY_URL"]
# Push at "https://${GITLAB_USER_NAME}:${PERSONAL_ACCESS_TOKEN}#gitlab.companyname.net/demo/demo.git")
# generatedPersonalAccessToken has been generated with full rights from https://gitlab.companyname.net/profile/personal_access_tokens and set in a variable not seen here
gitlabPushURL = "https://{}:{}#{}".format(gitlabUserLogin, generatedPersonalAccessToken, gitlabFullURL.split("#")[-1])
print("gitlabFullURL is [{}]".format(gitlabFullURL))
print("gitlabPushURL is [{}]".format(gitlabPushURL))
branchName = str(uuid.uuid1())
print("Build git.Repo object with [{}] root path".format(GIT_ROOT_PATH))
repo = git.Repo(GIT_ROOT_PATH)
print("Fetch all")
repo.git.fetch("-a")
print("Create new local branch [{}]".format(branchName))
repo.git.checkout("-b", branchName)
print("Modify file")
versionFile = os.path.join(GIT_ROOT_PATH, "public", "include" , "Version.h")
patchedVersionFileContent = ""
with open(versionFile, 'r') as versionFileContent:
patchedVersionFileContent = versionFileContent.read()
patchedVersionFileContent = re.sub("#define VERSION_MAJOR 0", "#define VERSION_MAJOR {}".format(75145), patchedVersionFileContent)
with open(versionFile, 'w') as versionFileContent:
versionFileContent.write(patchedVersionFileContent)
print("Stage file")
repo.git.add("-u")
print("Commit file")
repo.git.commit("-m", "New version file in new branch {}".format(branchName))
print("Push new branch [{}] remotely".format(branchName))
# The error is at below line:
repo.git.push(gitlabPushURL, "origin", branchName)
sys.exit(0)
except Exception as e:
print("Exception: {}".format(e))
sys.exit(-1)
Problem
Even with the trick to have rights, when we try to push from GitLab runner following error is raised:
Cmd('git') failed due to: exit code(1)
cmdline: git push https://user:token#gitlab.companyname.net/demo/repo.git origin 85a3fa6e-690a-11ea-a07d-e454e8696d31
stderr: 'error: src refspec origin does not match any
error: failed to push some refs to 'https://user:token#gitlab.companyname.net/demo/repo.git''
What works
If I open a Git Bash, I successfully run manual commands:
git fetch -a
git checkout -b newBranch
vim public/include/Version.h
=> At this point file has been modified
git add -u
git commit -m "New version file in new branch"
git push origin newBranch
Here if we fetch all from elsewhere we can see newBranch with version file modifications
And same if we run script content (without URL modification) from a python command line (assuming all imports as in script have been performed):
GIT_ROOT_PATH = "E:\\path\\to\\workspace\\repo"
branchName = str(uuid.uuid1())
repo = git.Repo(GIT_ROOT_PATH)
repo.git.fetch("-a")
repo.git.checkout("-b", branchName)
versionFile = os.path.join(GIT_ROOT_PATH, "public", "include" , "Version.h")
patchedVersionFileContent = ""
with open(versionFile, 'r') as versionFileContent:
patchedVersionFileContent = versionFileContent.read()
patchedVersionFileContent = re.sub("#define VERSION_MAJOR 0", "#define VERSION_MAJOR {}".format(75145), patchedVersionFileContent)
with open(versionFile, 'w') as versionFileContent:
versionFileContent.write(patchedVersionFileContent)
repo.git.add("-u")
repo.git.commit("-m", "New version file in new branch {}".format(branchName))
repo.git.push("origin", branchName)
Conclusion
I can't find what I do wrong when running from GitLab runner, is there something I'm missing ?
The only thing that I can see different when running from GitLab runner is that after fetch I can see I'm on a detached head (listing repo.git.branch('-a').split('\n') gives for example ['* (HEAD detached at 560976b)', 'branchName', 'remotes/origin/otherExistingBranch', ...]), but this should not be a problem since I create a new branch where to push, right ?

Git said that you used the wrong refspec. When you need to push in other remote you have to make it first gitlab = repo.create_remote("gitlab", gitlabPushURL) and push to it like repo.push("gitlab", branchName).
Edit from #gluttony to not break on next git run with "remote already exists":
remote_name = "gitlab"
if remote_name not in repo.remotes:
repo.create_remote(remote_name, gitlabPushURL)

Gitpython: test python pass but no file pushed in the repository

I'm about executing test using python and as selenium webdriver as a server , i added the gitpython module that i can apply a git test (clone , add , commit and push)
Every thing is Ok , and the test is passed but , when i check my repo i don't find the file that supposed pushed in the repository.
This is the code :
myrepo = Repo.clone_from(clonefield,local_repo_path,env={'GIT_SSH_COMMAND': ssh_cmd})
assert myrepo.__class__ is Repo
myfile = os.path.join(local_repo_path, 'new-file')
# This function just creates an empty file ...
open(myfile, 'wb').close()
myrepo.index.add([myfile])
myrepo.index.commit("initial commit")
# Setup a local tracking branch of a remote branch
print myrepo.remote().push("HEAD:refs/drafts/master")
This is the result :
No handlers could be found for logger "git.remote"
[git.remote.PushInfo object at 0x24fb9b0]
.
Ran 1 test in 88.865s
OK
Can any one help me to resolve this.
Thank you for your help.

I tried to fix it like this and it works good :)
myrepo = Repo.clone_from(clonefield,local_repo_path,env={'GIT_SSH_COMMAND': ssh_cmd})
assert myrepo.__class__ is Repo
myfile = os.path.join(local_repo_path, 'new-file')
# This function just creates an empty file ...
open(myfile, 'wb').close()
myrepo.index.add([myfile])
author = Actor("author", "author#example.com")
committer = Actor("committer", "committer#exapmle.com")
index.commit("my commit message", author=author, committer=committer)
# Setup a local tracking branch of a remote branch
myrepo.remotes.origin.push("HEAD:refs/drafts/master")[0].summary

Use bare repo with git-python

When I'm trying to add files to bare repo:
import git
r = git.Repo("./bare-repo")
r.working_dir("/tmp/f")
print(r.bare) # True
r.index.add(["/tmp/f/foo"]) # Exception, can't use bare repo <...>
I only understood that I can add files only by Repo.index.add.
Is using bare repo with git-python module even possible? Or I need to use subprocess.call with git --work-tree=... --git-dir=... add ?

You can not add files into bare repositories. They are for sharing, not for working. You should clone bare repository to work with it. There is a nice post about it: www.saintsjd.com/2011/01/what-is-a-bare-git-repository/
UPDATE (16.06.2016)
Code sample as requested:
import git
import os, shutil
test_folder = "temp_folder"
# This is your bare repository
bare_repo_folder = os.path.join(test_folder, "bare-repo")
repo = git.Repo.init(bare_repo_folder, bare=True)
assert repo.bare
del repo
# This is non-bare repository where you can make your commits
non_bare_repo_folder = os.path.join(test_folder, "non-bare-repo")
# Clone bare repo into non-bare
cloned_repo = git.Repo.clone_from(bare_repo_folder, non_bare_repo_folder)
assert not cloned_repo.bare
# Make changes (e.g. create .gitignore file)
tmp_file = os.path.join(non_bare_repo_folder, ".gitignore")
with open(tmp_file, 'w') as f:
f.write("*.pyc")
# Run git regular operations (I use cmd commands, but you could use wrappers from git module)
cmd = cloned_repo.git
cmd.add(all=True)
cmd.commit(m=".gitignore was added")
# Push changes to bare repo
cmd.push("origin", "master", u=True)
del cloned_repo # Close Repo object and cmd associated with it
# Remove non-bare cloned repo
shutil.rmtree(non_bare_repo_folder)

How to fetch using dulwich in python

I'm trying to do the equivalent of git fetch -a using the dulwich library within python.
Using the docs at https://www.dulwich.io/docs/tutorial/remote.html I created the following script:
from dulwich.client import LocalGitClient
from dulwich.repo import Repo
import os
home = os.path.expanduser('~')
local_folder = os.path.join(home, 'temp/local'
local = Repo(local_folder)
remote = os.path.join(home, 'temp/remote')
remote_refs = LocalGitClient().fetch(remote, local)
local_refs = LocalGitClient().get_refs(local_folder)
print(remote_refs)
print(local_refs)
with an existing git repository at ~/temp/remote and a newly initialised repo at ~/temp/local
remote_refs shows everything I would expect, but local_refs is an empty dictionary and git branch -a on the local repo returns nothing.
Am I missing something obvious?
This is on dulwich 0.12.0 and Python 3.5
EDIT #1
Following a discussion on the python-uk irc channel, I updated my script to include the use of determine_wants_all:
from dulwich.client import LocalGitClient
from dulwich.repo import Repo
home = os.path.expanduser('~')
local_folder = os.path.join(home, 'temp/local'
local = Repo(local_folder)
remote = os.path.join(home, 'temp/remote')
wants = local.object_store.determine_wants_all
remote_refs = LocalGitClient().fetch(remote, local, wants)
local_refs = LocalGitClient().get_refs(local_folder)
print(remote_refs)
print(local_refs)
but this had no effect :-(
EDIT #2
Again, following discussion on the python-uk irc channel, I tried running dulwich fetch from within the local repo. It gave the same result as my script i.e. the remote refs were printed to the console correctly, but git branch -a showed nothing.
EDIT - Solved
A simple loop to update the local refs did the trick:
from dulwich.client import LocalGitClient
from dulwich.repo import Repo
import os
home = os.path.expanduser('~')
local_folder = os.path.join(home, 'temp/local')
local = Repo(local_folder)
remote = os.path.join(home, 'temp/remote')
remote_refs = LocalGitClient().fetch(remote, local)
for key, value in remote_refs.items():
local.refs[key] = value
local_refs = LocalGitClient().get_refs(local_folder)
print(remote_refs)
print(local_refs)

LocalGitClient.fetch() does not update refs, it just fetches objects and then returns the remote refs so you can use that to update the target repository refs.

How to checkout a tag with GitPython

In a python script, I try to checkout a tag after cloning a git repository.
I use GitPython 0.3.2.
#!/usr/bin/env python
import git
g = git.Git()
g.clone("user#host:repos")
g = git.Git(repos)
g.execute(["git", "checkout", "tag_name"])
With this code I have an error:
g.execute(["git", "checkout", "tag_name"])
File "/usr/lib/python2.6/site-packages/git/cmd.py", line 377, in execute
raise GitCommandError(command, status, stderr_value)
GitCommandError: 'git checkout tag_name' returned exit status 1: error: pathspec 'tag_name' did not match any file(s) known to git.
If I replace the tag name with a branch name, I have no problem.
I didn't find informations in GitPython documentation.
And if I try to checkout the same tag in a shell, I have non problem.
Do you know how can I checkout a git tag in python ?

Assuming you cloned the repository in 'path/to/repo', just try this:
from git import Git
g = Git('path/to/repo')
g.checkout('tag_name')

from git import Git
g = Git(repo_path)
g.init()
g.checkout(version_tag)
Like cmd.py Class Git comments say
"""
The Git class manages communication with the Git binary.
It provides a convenient interface to calling the Git binary, such as in::
g = Git( git_dir )
g.init() # calls 'git init' program
rval = g.ls_files() # calls 'git ls-files' program
``Debugging``
Set the GIT_PYTHON_TRACE environment variable print each invocation
of the command to stdout.
Set its value to 'full' to see details about the returned values.
"""

git.Repo().git.checkout('tag')

This worked for me, and I think it's closer to the intended API usage:
from git import Repo
repo = Repo.clone_from("https://url_here", "local_path")
repo.heads['tag-name'].checkout()