Table of contents
Step 1: Generate documentation
Step 2: Trigger a pipeline in your website project
Step 3: Download the artifact to the website project
Step 4: Build and deploy the site
Using GitLab CI Multi Project Pipelines
This page contains instructions on how to leverage GitLab CI to run pipelines in multiple GitLab projects (repositories). This is particularly useful for projects that have multiple repositories and/or automatically generated documentation that should be kept up-to-date on the project website.
GitLab CI is a very flexible piece of software. As such, there is no One True Way of doing things. However, the recipe below is being successfully used by the cl-tar project to keep its website up-to-date with automatically generated documentation whenever it performs a release.
We will focus on the interaction between cl-tar/cl-tar-file and cl-tar/cl-tar.common-lisp.dev.
Step 1: Generate documentation
First, generate your documentation using GitLab CI. The relevant job from the cl-tar-file
project is:
# Build the docs and upload them to Gitlab.
generate docs:
extends:
- .clci sbcl
- .clci clpm script
variables:
CLCI_SCRIPT: scripts/generate-docs.lisp
artifacts:
paths:
- docs/
This job "extends" some helper jobs provided by the CLCI GitLab templates. The .clci sbcl
template says that this job should use SBCL. The .clci clpm script
template says that the latest version of ASDF should be installed, the latest version of CLPM should be installed, the dependencies specified in the CLPM lock file should be installed, and then the file pointed to by $CLCI_SCRIPT
should be cl:load
ed into an SBCL process where ASDF has been loaded and all the dependencies are availble.
The scripts/generate-docs.lisp
file uses a CL-based documentation generator to output docs to the docs
folder.
Last, the docs
folder is uploaded to GitLab as an artifact.
Step 2: Trigger a pipeline in your website project
Next, you must trigger a job in your project's website's repo. The relevant job from the cl-tar-file
project is:
# Notify the project containing the website that new documentation is ready to
# be fetched.
publish docs:
rules:
- if: '$CI_COMMIT_TAG =~ /^v[0-9]+(\.[0-9]+)*(-.*)?$/'
needs:
- "blocker:release:clci"
- "generate docs"
trigger: cl-tar/cl-tar.common-lisp.dev
variables:
CL_TAR_FILE_TAG: $CI_COMMIT_TAG
CL_TAR_FILE_PIPELINE: $CI_PIPELINE_ID
This job's rules
ensure that it is only run when a tag is pushed to the repo that looks like a version number.
The needs
ensures that this job is not run until both the documentation has actually been generated and all sanity checks performed by the CLCI GitLab template release pipeline have finished. These sanity checks include making sure the tag is protected and all tests pass.
The trigger
says that a new pipeline should be trigged on the default branch of the cl-tar/cl-tar.common-lisp.dev
project.
Last, the variables
section passes the pipeline ID and tag to the newly created pipeline.
Step 3: Download the artifact to the website project
Next, the newly triggered pipeline downloads and commits the generated documentation. The relevant jobs from the cl-tar.common-lisp.dev
project are this one from .gitlab-ci.yml
:
download cl-tar-file docs:
variables:
CL_TAR_FILE_PIPELINE_ID: $CL_TAR_FILE_PIPELINE
CL_TAR_FILE_TAG: $CL_TAR_FILE_TAG
rules:
- if: $CL_TAR_FILE_PIPELINE
trigger:
include: .gitlab-ci.cl-tar-file.yml
strategy: depend
And this one from .gitlab-ci.cl-tar-file.yml
:
download docs:
image: alpine:3.13
script:
- apk add jq unzip git curl
- git config --global user.email "$GITLAB_USER_EMAIL"
- git config --global user.name "$GITLAB_USER_NAME"
- git checkout $CI_COMMIT_BRANCH
- git reset --hard origin/$CI_COMMIT_BRANCH
# Find the job id of the docs job
- JOB_ID="$(curl -fsSL "$CI_API_V4_URL/projects/1508/pipelines/$CL_TAR_FILE_PIPELINE_ID/jobs" | jq '.[] | select (.name == "generate docs") | .id')"
# Download the artifact.zip
- curl -fsSL "https://gitlab.common-lisp.net/cl-tar/cl-tar-file/-/jobs/$JOB_ID/artifacts/download?file_type=archive" > artifacts.zip
- unzip artifacts.zip
- mkdir -p cl-tar-file
- mv docs cl-tar-file/$CL_TAR_FILE_TAG
- git add cl-tar-file/$CL_TAR_FILE_TAG
- git commit -m "cl-tar-file $CL_TAR_FILE_TAG docs"
- git push https://git:$UPDATER_ACCESS_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git $CI_COMMIT_BRANCH:$CI_COMMIT_BRANCH
The first job simple creates a child pipeline from the .gitlab-ci.cl-tar-file.yml
if certain conditions are met (namely, the variable $CL_TAR_FILE_PIPELINE
is set). This child pipeline is completely optional: the following job could have been written in .gitlab-ci.yml
with appropriate rules
added.
The second job takes the pipeline ID that generated the documentation, queries the GitLab API to find the URL for the artifacts, and then downloads the artifacts. It then unzips them and copies the folder to the correct place in the repo. Last, it commits it and pushes.
Step 4: Build and deploy the site
After the push from Step 3, another pipeline is kicked off. This time, the new documentation from the cl-tar-file
project is present in the repo. This triggers the following job in cl-tar.common-lisp.dev
, which in turn ends up building the website and deploying it using GitLab Pages.
publish docs:
rules:
- if: '$CL_TAR_PIPELINE == null && $CL_TAR_FILE_PIPELINE == null'
trigger:
include: .gitlab-ci.publish.yml
strategy: depend