Photo by Remy Gieling on Unsplash.

I really like standards and processes ensuring a good development workflow for my projects. But I work alone on this website, on my free time. Automating is therefore a key aspect so that I can deploy quickly when working on this project.

I already talked about optimisations of GitLab CI pipelines, but this time I wanted to question the whole deployment workflow. It was already automated by CI/CD:

update some files with the new version number
merge in master
create a Git tag
push everything to GitLab to trigger a deployment pipeline

Everything was done by running a simple Bash script locally, so a bit fragile. The main goal is to replace this script by the pipeline using Git branches and Merge Requests.

Defining the deployment workflow
Practical part
Conclusion

Defining the deployment workflow

GitFlow for Git branches

If « GitFlow » doesn't mean anything to you, it's a popular Git branching model introduced in 2010 in a blog post by Vincent Driessen: A successful Git branching model.

To sum up GitFlow, the master branch describes the current state of your production environment and contains Git tags for each version. The version under development is on the develop branch and every new feature is done on a feature/... branch.
When a feature is finished it is merged in develop. Then to deploy develop we create a release/... branch from it, which will be merged into master. We finally create a tag on master.

If you discover GitFlow I recommend reading the Atlassian guide about it which illustrate the branching model and addresses other aspects ommited in this post.

Why use GitFlow ? This model has proven itself and suits very well to applications development (here a website). It also became a habit for many developers.
My current workflow does include the develop branch but it gets direcly merged into master without any release branch. Those are very useful while working in a team but not a lot for solo projects.

Automatic deployment steps

The original desired workflow is as follows:

A feature is finished, a Merge Request is opened towards develop which triggers a CI pipeline.
We then merge the MR into develop which triggers a new pipeline.
This pipeline will check if a new version has been declared for our app.
In this case, we update a few files if needed and create a MR for the desired branch (master in my case).

Every step must be automated except for merging the MR.
But there's one thing to solve: how to check if a new version of the app has been declared ?

Changelog to the rescue

A changelog is a file containing every changes in a project for every version. It is commonly generated from the Git logs but we're going to do the exact opposite: generate new versions from the changelog !

I already maintain a changelog for a long time based on Keep A Changelog in MarkDown. Unreleased changes have their own section at the top, and each version is declared in a title as follows:

## [v1.2.3] - 2021-05-28

« Nice, but how does this help us ? »

If we want to deliver a new version, we declare it in the CHANGELOG.md file. In the pipeline we can extract the last declared version of the changelog and compare it to the last Git tag. If those values are different then we must deploy !

Practical part

Creating a token for automated operations

Some operations like writing to a Git repository are forbidden using the default credentials provided by GitLab CI pipelines. We must create a Personal Access Token. Simply follow the GitLab documentation: name the token et choose the api and write_repository scopes.

To securely use this token we will save it as a CI/CD variable: name the variable as you will use it in the pipeline, give the token as its value and check the box to mask the variable in CI logs.
Mine will be called AYMDEV_GITLAB_TOKEN.

Git environment

As we need to use git commands we'll use the bitnami/git image. The name will be saved into a pipeline variable in .gitlab-ci.yml:

variables:
    # ...
    GIT_IMAGE: "bitnami/git:2.33.0-debian-10-r61"

Every job using git commands will use this image but will also need to configure Git. To reuse the following commands we'll declare a YAML anchor:

.git-setup: &git-setup
    - git config --global user.name "${GITLAB_USER_LOGIN}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git remote set-url origin "https://${GITLAB_USER_LOGIN}:${AYMDEV_GITLAB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
    - git tag -l | xargs git tag -d
    - git fetch --tags

Here we use predefined environment variables and the token variable to give Git a minimal configuration. Tags get synchronized to avoid blocking the pipeline in case of any error.

Extracting version numbers from the changelog

We'll use a RegEx to extract the version numbers. Be careful though because there are many RegEx engines and they're not all available natively in the Docker images used in CI. I really enjoy the PCRE engine (Perl Compatible Regular Expression) but perl isn't always installed, and grep's -P option isn't always available.
So we'll use grep -E with a POSIX-Extended RegEx, and the -o option to only get the part that matches the pattern.
There are many inconvenients using this kind of RegEx: we can't use shortcuts like \d, nor capture groups as we would do in PHP.

Let's start by retrieving title lines containing version numbers:

^## \[v[0-9]+\.[0-9]+\.[0-9]+\] - [0-9]{4}(-[0-9]{2}){2}$

We begin with the start of the line (^), the title, version number, creation date and the end of the line ($).
Result: ## [v1.2.3] - 2021-05-28

We pass this result to a second expression which will only extract the version number:

v[0-9]+\.[0-9]+\.[0-9]+

We only get the part which is between the square brackets [].
Result: v1.2.3

We only keep the first result as the last declared version. To easily use this value we declare a new YAML anchor that will declare a CURRENT_VERSION variable for the jobs that will need it:

.get-git-tag: &get-git-tag
    - export CURRENT_VERSION=$(grep -oE '^## \[v[0-9]+\.[0-9]+\.[0-9]+\] - [0-9]{4}(-[0-9]{2}){2}$' CHANGELOG.md | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -n 1)

Merging of a Merge Request into `develop`

A pipeline for the develop branch is triggered as a MR gets merged. We'll add a job at the end to check if a new version has been declared:

trigger-merge-request:
    stage: release
    image: "$GIT_IMAGE"
    rules:
        - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "develop"'
    before_script:
        - *git-setup
        - *get-git-tag
        - git checkout develop
        - git reset --hard origin/develop
    script:
        - ./ci/merge-request.sh

We use the git-setup and get-git-tag anchors, synchronize the local branch with the remote repository, and then run a Bash script ci/merge-request.sh with the following content:

#!/usr/bin/env bash

# The job must fail at any error
set -xe

# Retrieval of the last Git tag
LAST_GIT_TAG=$(git tag | sort -V | tail -1)

# We stop if it's the same as the last changelog version
if [ "$LAST_GIT_TAG" == "$CURRENT_VERSION" ]; then
    exit 0
fi


# Update of a few files using the version number
sed -i "s/${LAST_GIT_TAG}/${CURRENT_VERSION}/g" README.md
sed -i "s~${CI_REGISTRY_IMAGE}:${LAST_GIT_TAG}~${CI_REGISTRY_IMAGE}:${CURRENT_VERSION}~" docker-compose.yml


# If they changed, we create and push a commit to GitLab
if [ "$(git diff --exit-code README.md docker-compose.prod.yml)" ]; then
	git add README.md docker-compose.prod.yml
	git commit -m "UPGRADE app to ${CURRENT_VERSION}"
	git push -o ci.skip
fi


# JSON content to send to the GitLab API to create a Merge Request
REQUEST_BODY=$(cat <<-JSON
	{
	    "source_branch": "develop",
	    "target_branch": "master",
	    "title": "[AUTO-RELEASE] Deploy ${CURRENT_VERSION}"
	}
JSON
)

# Sending of a POST request to the GitLab API to create the Merge Request
curl -X POST "https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests" \
    --header "PRIVATE-TOKEN:${AYMDEV_GITLAB_TOKEN}" \
    --header "Content-Type: application/json" \
    --data "${REQUEST_BODY}"

When done we should see a new Merge Request on GitLab.

Note the command to push the commit: git push -o ci.skip.
ci.skip is a push option of GitLab which prevents triggering a pipeline. We don't want to trigger one for this commit which only changes the version number while the current develop pipeline passes.
However we need to configure the project to consider "skipped pipelines" valid in order to merge our new MR into master. In the UI, go to Settings > General > Merge Requests > Merge checks and check « Skipped pipelines are considered successful ».

Merging the Merge Request into `master`

The deployment pipeline is now running. We just need to add a job to create a Git tag to officially release the new version:

publish-git-tag:
    stage: release
    image: "$GIT_IMAGE"
    rules:
        - if: '$CI_COMMIT_BRANCH == "master"'
    before_script:
        - *git-setup
        - *get-git-tag
        - git checkout master
        - git reset --hard origin/master
    script:
        - git tag -a -m "Releasing app $CURRENT_VERSION" "$CURRENT_VERSION"
        - git push --tags

There is no prerequisite for the other jobs. On the other hand we can still use the changlog to extract the previous version to set a Docker image tag as the cache source to build a new image:

build-docker-image:
    stage: release
    rules:
        - if: '$CI_COMMIT_BRANCH == "master"'
    before_script:
        # ...
        - *get-git-tag
    script:
        - export PREVIOUS_VERSION=$(grep -oE '^## \[v[0-9]+\.[0-9]+\.[0-9]+\] - [0-9]{4}(-[0-9]{2}){2}$' CHANGELOG.md | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | sed -n '2 p')
        - docker build --pull --cache-from "$CI_REGISTRY_IMAGE:$PREVIOUS_VERSION" -t "$CI_REGISTRY_IMAGE:$CURRENT_VERSION" -f ./Dockerfile .
        - docker push "$CI_REGISTRY_IMAGE:$CURRENT_VERSION"

Conclusion

Releasing a new version is now almost automatic, we only need to declare a new version in the changelog !
This first approach is a base to adapt depending on the projects. In a development team it will probably be useful to create release branches instead of creating a MR to master. But we did the main part of the job, we know how to create commits, tags and Merge Requests from the pipeline !

On my side it will take a few more iterations before making this workflow really smooth, it's still too long and manual. As Vincent Driessen noted in his GitFlow post in 2020, if the project uses continuous deployment then a simpler branching model like the GitHub flow would be preferable.
Next goal ? Getting rid of the develop branch to only keep the features and master.

Deploy from a changelog in GitLab CI

Defining the deployment workflow

GitFlow for Git branches

Automatic deployment steps

Changelog to the rescue

Practical part

Creating a token for automated operations

Git environment

Extracting version numbers from the changelog

Merging of a Merge Request into `develop`

Merging the Merge Request into `master`

Conclusion

Comments: 0

Robot invasion coming from robohash.org

Deploy from a changelog in GitLab CI

# Defining the deployment workflow

# GitFlow for Git branches

# Automatic deployment steps

# Changelog to the rescue

# Practical part

# Creating a token for automated operations

# Git environment

# Extracting version numbers from the changelog

# Merging of a Merge Request into develop

# Merging the Merge Request into master

# Conclusion

Comments: 0

Robot invasion coming from robohash.org

Defining the deployment workflow

GitFlow for Git branches

Automatic deployment steps

Changelog to the rescue

Practical part

Creating a token for automated operations

Git environment

Extracting version numbers from the changelog

Merging of a Merge Request into `develop`

Merging the Merge Request into `master`

Conclusion