Table of contents

Step 1: Generate documentation

Step 2: Trigger a pipeline in your website project

Step 3: Download the artifact to the website project

Step 4: Build and deploy the site

Using GitLab CI Multi Project Pipelines

This page contains instructions on how to leverage GitLab CI to run pipelines in multiple GitLab projects (repositories). This is particularly useful for projects that have multiple repositories and/or automatically generated documentation that should be kept up-to-date on the project website.

GitLab CI is a very flexible piece of software. As such, there is no One True Way of doing things. However, the recipe below is being successfully used by the cl-tar project to keep its website up-to-date with automatically generated documentation whenever it performs a release.

We will focus on the interaction between cl-tar/cl-tar-file and cl-tar/

Step 1: Generate documentation

First, generate your documentation using GitLab CI. The relevant job from the cl-tar-file project is:

# Build the docs and upload them to Gitlab.  
generate docs:  
    - .clci sbcl  
    - .clci clpm script  
    CLCI_SCRIPT: scripts/generate-docs.lisp  
      - docs/ 

This job "extends" some helper jobs provided by the CLCI GitLab templates. The .clci sbcl template says that this job should use SBCL. The .clci clpm script template says that the latest version of ASDF should be installed, the latest version of CLPM should be installed, the dependencies specified in the CLPM lock file should be installed, and then the file pointed to by $CLCI_SCRIPT should be cl:loaded into an SBCL process where ASDF has been loaded and all the dependencies are availble.

The scripts/generate-docs.lisp file uses a CL-based documentation generator to output docs to the docs folder.

Last, the docs folder is uploaded to GitLab as an artifact.

Step 2: Trigger a pipeline in your website project

Next, you must trigger a job in your project's website's repo. The relevant job from the cl-tar-file project is:

# Notify the project containing the website that new documentation is ready to  
# be fetched.  
publish docs:  
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+(\.[0-9]+)*(-.*)?$/'  
    - "blocker:release:clci"  
    - "generate docs"  
  trigger: cl-tar/  

This job's rules ensure that it is only run when a tag is pushed to the repo that looks like a version number.

The needs ensures that this job is not run until both the documentation has actually been generated and all sanity checks performed by the CLCI GitLab template release pipeline have finished. These sanity checks include making sure the tag is protected and all tests pass.

The trigger says that a new pipeline should be trigged on the default branch of the cl-tar/ project.

Last, the variables section passes the pipeline ID and tag to the newly created pipeline.

Step 3: Download the artifact to the website project

Next, the newly triggered pipeline downloads and commits the generated documentation. The relevant jobs from the project are this one from .gitlab-ci.yml:

download cl-tar-file docs:  
    strategy: depend 

And this one from

download docs:  
  image: alpine:3.13  
    - apk add jq unzip git curl  
    - git config --global "$GITLAB_USER_EMAIL"  
    - git config --global "$GITLAB_USER_NAME"  
    - git checkout $CI_COMMIT_BRANCH  
    - git reset --hard origin/$CI_COMMIT_BRANCH  
    # Find the job id of the docs job  
    - JOB_ID="$(curl -fsSL "$CI_API_V4_URL/projects/1508/pipelines/$CL_TAR_FILE_PIPELINE_ID/jobs" | jq '.[] | select (.name == "generate docs") | .id')"  
    # Download the  
    - curl -fsSL "$JOB_ID/artifacts/download?file_type=archive" >  
    - unzip  
    - mkdir -p cl-tar-file  
    - mv docs cl-tar-file/$CL_TAR_FILE_TAG  
    - git add cl-tar-file/$CL_TAR_FILE_TAG  
    - git commit -m "cl-tar-file $CL_TAR_FILE_TAG docs"  

The first job simple creates a child pipeline from the if certain conditions are met (namely, the variable $CL_TAR_FILE_PIPELINE is set). This child pipeline is completely optional: the following job could have been written in .gitlab-ci.yml with appropriate rules added.

The second job takes the pipeline ID that generated the documentation, queries the GitLab API to find the URL for the artifacts, and then downloads the artifacts. It then unzips them and copies the folder to the correct place in the repo. Last, it commits it and pushes.

Step 4: Build and deploy the site

After the push from Step 3, another pipeline is kicked off. This time, the new documentation from the cl-tar-file project is present in the repo. This triggers the following job in, which in turn ends up building the website and deploying it using GitLab Pages.

publish docs:  
    - if: '$CL_TAR_PIPELINE == null && $CL_TAR_FILE_PIPELINE == null'  
    include: .gitlab-ci.publish.yml  
    strategy: depend