Feb 8, 2016

Templates in OpenStack's Zuul

This is a followup to my post  "Creating new test jobs in OpenStack CI". Last time I covered the basic setup of jobs by Jenkins and Zuul. Since many OpenStack projects run the same jobs, the Zuul developers have introduced templates to easily group and reuse these jobs.

Let's look at one common example, it's the python-jobs template. Like all examples in this article, it is defined in file zuul/layout.yaml in the openstack-infra/project-config repository and to use it, you need to edit the same file.

Defining your own templates

Since you now know how to use a template, let's explain how they really look like. A template consists of a name, definitions for which jobs should run in which queue and allows to substitute the name of the repository in the job, so {name} gets replaced by your repository, in the example by amazing-repo.

Let's look at the python-jobs template:



  - name: python-jobs
    check:
      - 'gate-{name}-pep8'
      - 'gate-{name}-docs'
      - 'gate-{name}-python27'
    gate:
      - 'gate-{name}-docs'
      - 'gate-{name}-pep8'
      - 'gate-{name}-python27'
    post:
      - '{name}-branch-tarball'


The template has the name python-jobs, adds three jobs to the check queue and the same jobs also to the gate queue. An additional job is added to the post queue. Jobs in the check get queue get triggered when a change gets submitted, jobs in the gate queue get triggered when a change gets approved by a core reviewer and jobs in the post queue get triggered after a change has merged.
If you are adding the same class of jobs to several repositories, create a template for it. A template can contain of a single job that is associated with one queue, or contain several jobs in several queues like the example above.

Using a template

So, if your project amazing-project wants to reuse the python-jobs template as is, just add it as template:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: python-jobs

You can also limit, on which branches those are jobs are triggered. For example, to run the docs job only on stable/liberty and newer branches, you can add a condition:

  - name: gate-amazing-project-docs
    branch: ^(?!stable/kilo).*$



So, instead of saying run on liberty and newer, we block it on older supported branches, in this case kilo is the only older supported branch.

If you're introducing jobs, best practice is to add them first to the experimental queue, and then add them as non-voting, and only finally as voting. In this case, the templates do not help you at all for the first two steps, you have to look at their definition and add them manually.

First step, using the jobs in the experimental queue:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: noop-jobs
    experimental:
      - gate-amazing-repo-pep8
      - gate-amazing-repo-docs
      - gate-amazing-repo-python27


Note that we use noop-jobs as a template, so that both check and gate queue have at least one job. The noop jobs do nothing but are important since Zuul requires at least one job to run with success, otherwise you will not be able to merge anything.

With this definition, you can now submit a change and add as review comment "check experimental" and the jobs are run and the results are reported.

Later, the manually triggered jobs run fine, so it's time to run them on each change but keep them non-voting to not block any merges:

  - name: gate-amazing-repo-docs
    voting: false

  - name: gate-amazing-repo-pep8
    voting: false

  - name: gate-amazing-repo-python27
    voting: false
....

  - name: openstack/amazing-repo
    template:
      - name: merge-check
    check:
      - gate-amazing-repo-pep8
      - gate-amazing-repo-docs
      - gate-amazing-repo-python27
    gate:
      - noop

Here we added the noop job to the gate since otherwise no job would run in the gate and Zuul requires at least one job to run.

Once the jobs all run fine, you can add them to the gate as well - and for that case, let's finally use the template:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: python-jobs



Defining your own templates

Since you now know how to use a template, let's explain how they really look like. A template consists of a name, definitions for which jobs should run in which queue and allows to substitute the name of the repository in the job, so {name} gets replaced by your repository, in the example by amazing-repo.
Let's review the python-jobs template again:

 
  - name: python-jobs
    check:
      - 'gate-{name}-pep8'
      - 'gate-{name}-docs'
      - 'gate-{name}-python27'
    gate:
      - 'gate-{name}-docs'
      - 'gate-{name}-pep8'
      - 'gate-{name}-python27'
    post:
      - '{name}-branch-tarball'


The template has the name python-jobs, adds three jobs to the check queue and the same jobs also to the gate queue. An additional job is added to the post queue. Jobs in the check get queue get triggered when a change gets submitted, jobs in the gate queue get triggered when a change gets approved by a core reviewer and jobs in the post queue get triggered after a change has merged.
If you are adding the same class of jobs to several repositories, create a template for it. A template can contain of a single job that is associated with one queue, or contain several jobs in several queues like the example above.

References

For more information about templates, you can look at the file zuul/layout.yaml on definitions and usage.  Zuul has been written for OpenStack CI and has its own documentation. For information about Zuul's OpenStack instance, read the Project Config Infrastructure page about Zuul. The best starting place learn about using the OpenStack CI infrastructure is the Infra Manual.

Followup?

If you liked this post and like to learn more about OpenStack CI, please leave a comment with details.