Using GitLab CI Multi Project Pipelines

This page contains instructions on how to leverage GitLab CI to run pipelines in multiple GitLab projects (repositories). This is particularly useful for projects that have multiple repositories and/or automatically generated documentation that should be kept up-to-date on the project website.

GitLab CI is a very flexible piece of software. As such, there is no One True Way of doing things. However, the recipe below is being successfully used by the cl-tar project to keep its website up-to-date with automatically generated documentation whenever it performs a release.

We will focus on the interaction between cl-tar/cl-tar-file and cl-tar/cl-tar.common-lisp.dev.

Step 1: Generate documentation

First, generate your documentation using GitLab CI. The relevant job from the cl-tar-file project is:

# Build the docs and upload them to Gitlab.  
generate docs:  
  extends:  
    - .clci sbcl  
    - .clci clpm script  
  variables:  
    CLCI_SCRIPT: scripts/generate-docs.lisp  
  artifacts:  
    paths:  
      - docs/ 

This job "extends" some helper jobs provided by the CLCI GitLab templates. The .clci sbcl template says that this job should use SBCL. The .clci clpm script template says that the latest version of ASDF should be installed, the latest version of CLPM should be installed, the dependencies specified in the CLPM lock file should be installed, and then the file pointed to by $CLCI_SCRIPT should be cl:loaded into an SBCL process where ASDF has been loaded and all the dependencies are availble.

The scripts/generate-docs.lisp file uses a CL-based documentation generator to output docs to the docs folder.

Last, the docs folder is uploaded to GitLab as an artifact.

Step 2: Trigger a pipeline in your website project

Next, you must trigger a job in your project's website's repo. The relevant job from the cl-tar-file project is:

# Notify the project containing the website that new documentation is ready to  
# be fetched.  
publish docs:  
  rules:  
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+(\.[0-9]+)*(-.*)?$/'  
  needs:  
    - "blocker:release:clci"  
    - "generate docs"  
  trigger: cl-tar/cl-tar.common-lisp.dev  
  variables:  
    CL_TAR_FILE_TAG: $CI_COMMIT_TAG  
    CL_TAR_FILE_PIPELINE: $CI_PIPELINE_ID 

This job's rules ensure that it is only run when a tag is pushed to the repo that looks like a version number.

The needs ensures that this job is not run until both the documentation has actually been generated and all sanity checks performed by the CLCI GitLab template release pipeline have finished. These sanity checks include making sure the tag is protected and all tests pass.

The trigger says that a new pipeline should be trigged on the default branch of the cl-tar/cl-tar.common-lisp.dev project.

Last, the variables section passes the pipeline ID and tag to the newly created pipeline.

Step 3: Download the artifact to the website project

Next, the newly triggered pipeline downloads and commits the generated documentation. The relevant jobs from the cl-tar.common-lisp.dev project are this one from .gitlab-ci.yml:

download cl-tar-file docs:  
  variables:  
    CL_TAR_FILE_PIPELINE_ID: $CL_TAR_FILE_PIPELINE  
    CL_TAR_FILE_TAG: $CL_TAR_FILE_TAG  
  rules:  
    - if: $CL_TAR_FILE_PIPELINE  
  trigger:  
    include: .gitlab-ci.cl-tar-file.yml  
    strategy: depend 

And this one from .gitlab-ci.cl-tar-file.yml:

download docs:  
  image: alpine:3.13  
  script:  
    - apk add jq unzip git curl  
    - git config --global user.email "$GITLAB_USER_EMAIL"  
    - git config --global user.name "$GITLAB_USER_NAME"  
    - git checkout $CI_COMMIT_BRANCH  
    - git reset --hard origin/$CI_COMMIT_BRANCH  
    # Find the job id of the docs job  
    - JOB_ID="$(curl -fsSL "$CI_API_V4_URL/projects/1508/pipelines/$CL_TAR_FILE_PIPELINE_ID/jobs" | jq '.[] | select (.name == "generate docs") | .id')"  
    # Download the artifact.zip  
    - curl -fsSL "https://gitlab.common-lisp.net/cl-tar/cl-tar-file/-/jobs/$JOB_ID/artifacts/download?file_type=archive" > artifacts.zip  
    - unzip artifacts.zip  
    - mkdir -p cl-tar-file  
    - mv docs cl-tar-file/$CL_TAR_FILE_TAG  
    - git add cl-tar-file/$CL_TAR_FILE_TAG  
    - git commit -m "cl-tar-file $CL_TAR_FILE_TAG docs"  
    - git push https://git:$UPDATER_ACCESS_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git $CI_COMMIT_BRANCH:$CI_COMMIT_BRANCH 

The first job simple creates a child pipeline from the .gitlab-ci.cl-tar-file.yml if certain conditions are met (namely, the variable $CL_TAR_FILE_PIPELINE is set). This child pipeline is completely optional: the following job could have been written in .gitlab-ci.yml with appropriate rules added.

The second job takes the pipeline ID that generated the documentation, queries the GitLab API to find the URL for the artifacts, and then downloads the artifacts. It then unzips them and copies the folder to the correct place in the repo. Last, it commits it and pushes.

Step 4: Build and deploy the site

After the push from Step 3, another pipeline is kicked off. This time, the new documentation from the cl-tar-file project is present in the repo. This triggers the following job in cl-tar.common-lisp.dev, which in turn ends up building the website and deploying it using GitLab Pages.

publish docs:  
  rules:  
    - if: '$CL_TAR_PIPELINE == null && $CL_TAR_FILE_PIPELINE == null'  
  trigger:  
    include: .gitlab-ci.publish.yml  
    strategy: depend