Table of contents
Using GitLab CI Multi Project Pipelines
This page contains instructions on how to leverage GitLab CI to run pipelines in multiple GitLab projects (repositories). This is particularly useful for projects that have multiple repositories and/or automatically generated documentation that should be kept up-to-date on the project website.
GitLab CI is a very flexible piece of software. As such, there is no One True Way of doing things. However, the recipe below is being successfully used by the cl-tar project to keep its website up-to-date with automatically generated documentation whenever it performs a release.
We will focus on the interaction between cl-tar/cl-tar-file and cl-tar/cl-tar.common-lisp.dev.
Step 1: Generate documentation
First, generate your documentation using GitLab CI. The relevant job from the
cl-tar-file project is:
# Build the docs and upload them to Gitlab. generate docs: extends: - .clci sbcl - .clci clpm script variables: CLCI_SCRIPT: scripts/generate-docs.lisp artifacts: paths: - docs/
This job "extends" some helper jobs provided by the CLCI GitLab templates. The
.clci sbcl template says that this job should use SBCL. The
.clci clpm script template says that the latest version of ASDF should be installed, the latest version of CLPM should be installed, the dependencies specified in the CLPM lock file should be installed, and then the file pointed to by
$CLCI_SCRIPT should be
cl:loaded into an SBCL process where ASDF has been loaded and all the dependencies are availble.
scripts/generate-docs.lisp file uses a CL-based documentation generator to output docs to the
docs folder is uploaded to GitLab as an artifact.
Step 2: Trigger a pipeline in your website project
Next, you must trigger a job in your project's website's repo. The relevant job from the
cl-tar-file project is:
# Notify the project containing the website that new documentation is ready to # be fetched. publish docs: rules: - if: '$CI_COMMIT_TAG =~ /^v[0-9]+(\.[0-9]+)*(-.*)?$/' needs: - "blocker:release:clci" - "generate docs" trigger: cl-tar/cl-tar.common-lisp.dev variables: CL_TAR_FILE_TAG: $CI_COMMIT_TAG CL_TAR_FILE_PIPELINE: $CI_PIPELINE_ID
rules ensure that it is only run when a tag is pushed to the repo that looks like a version number.
needs ensures that this job is not run until both the documentation has actually been generated and all sanity checks performed by the CLCI GitLab template release pipeline have finished. These sanity checks include making sure the tag is protected and all tests pass.
trigger says that a new pipeline should be trigged on the default branch of the
variables section passes the pipeline ID and tag to the newly created pipeline.
Step 3: Download the artifact to the website project
Next, the newly triggered pipeline downloads and commits the generated documentation. The relevant jobs from the
cl-tar.common-lisp.dev project are this one from
download cl-tar-file docs: variables: CL_TAR_FILE_PIPELINE_ID: $CL_TAR_FILE_PIPELINE CL_TAR_FILE_TAG: $CL_TAR_FILE_TAG rules: - if: $CL_TAR_FILE_PIPELINE trigger: include: .gitlab-ci.cl-tar-file.yml strategy: depend
And this one from
download docs: image: alpine:3.13 script: - apk add jq unzip git curl - git config --global user.email "$GITLAB_USER_EMAIL" - git config --global user.name "$GITLAB_USER_NAME" - git checkout $CI_COMMIT_BRANCH - git reset --hard origin/$CI_COMMIT_BRANCH # Find the job id of the docs job - JOB_ID="$(curl -fsSL "$CI_API_V4_URL/projects/1508/pipelines/$CL_TAR_FILE_PIPELINE_ID/jobs" | jq '. | select (.name == "generate docs") | .id')" # Download the artifact.zip - curl -fsSL "https://gitlab.common-lisp.net/cl-tar/cl-tar-file/-/jobs/$JOB_ID/artifacts/download?file_type=archive" > artifacts.zip - unzip artifacts.zip - mkdir -p cl-tar-file - mv docs cl-tar-file/$CL_TAR_FILE_TAG - git add cl-tar-file/$CL_TAR_FILE_TAG - git commit -m "cl-tar-file $CL_TAR_FILE_TAG docs" - git push https://git:$UPDATER_ACCESS_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git $CI_COMMIT_BRANCH:$CI_COMMIT_BRANCH
The first job simple creates a child pipeline from the
.gitlab-ci.cl-tar-file.yml if certain conditions are met (namely, the variable
$CL_TAR_FILE_PIPELINE is set). This child pipeline is completely optional: the following job could have been written in
.gitlab-ci.yml with appropriate
The second job takes the pipeline ID that generated the documentation, queries the GitLab API to find the URL for the artifacts, and then downloads the artifacts. It then unzips them and copies the folder to the correct place in the repo. Last, it commits it and pushes.
Step 4: Build and deploy the site
After the push from Step 3, another pipeline is kicked off. This time, the new documentation from the
cl-tar-file project is present in the repo. This triggers the following job in
cl-tar.common-lisp.dev, which in turn ends up building the website and deploying it using GitLab Pages.
publish docs: rules: - if: '$CL_TAR_PIPELINE == null && $CL_TAR_FILE_PIPELINE == null' trigger: include: .gitlab-ci.publish.yml strategy: depend