Jan 19, 2014

Setting up gating in the OpenStack intrastructure

One thing that impressed me with OpenStack, is the common gerrit workflow that is not only used for code but also for documentation and the infrastructure.
As I wrote in my last blog post, I enabled further validation of the documentation.
This article will describe how the validation has been done using the OpenStack CI infrastructure.

The validation is done via Jenkins and to execute a process during review, as part of validation after it is approved, after it is merged or at periodic times, you need to create jobs for Jenkins and schedule them via Zuul, the OpenStack project gating system. Let's look at an example how this can be setup.

An example of gating

Let's look at how I setup gating for the api-sites projects. The patch I sent is https://review.openstack.org/#/c/64795/ and will be used as example in this blog post.
First, you should learn about the OpenStack Continous Integration infrastructure which is documented at http://ci.openstack.org/.  Then you should check out the infra repository called config since it contains all configuration files you need to modify. Instead of directly editing Jenkins configuration, YAML files are written that the Jenkins Job Builder then uses to create Jenkins jobs.

Defining a job

For a simple job, you can just specify it. With openstack-doc-tools, I wanted to reuse the same job not only for the api-sites but for all other projects as well. This can be done with a job template and those can be grouped as job-groups. Since my jobs can be executed by tox, I could reuse the 'gate-{name}-tox-{envlist}' template (file modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml). Parameters are specified in curly braces, thus it needs as parameters the "name" of the job and the "envlist" that tox will run - basically via "tox -e envlist".
To save typing, I used job-groups to define all four jobs in one places in the file modules/openstack_project/files/jenkins_job_builder/config/manuals-jobs.yaml:
 - job-group:
    name: openstack-doc-jobs
    jobs:
      - gate-{name}-tox-{envlist}:
          envlist: checkniceness
      - gate-{name}-tox-{envlist}:
          envlist: checksyntax
      - gate-{name}-tox-{envlist}:
          envlist: checkdeletions
      - gate-{name}-tox-{envlist}:
          envlist: checkbuild

So, this creates 'gate-{name}-tox-checkniceness' etc, the variable envlist gets substituted with checkniceness for the first gate etc.

Now, to use this job-group, I added to modules/openstack_project/files/jenkins_job_builder/config/projects.yaml a project definition:
- project:
    name: api-site
    github-org: openstack
    node: precise

    jobs:
      - openstack-doc-jobs

The name parameter in the job-group is now substituted with "api-site", thus this creates the four jobs:
gate-api-site-tox-checkniceness
gate-api-site-tox-checksyntax
gate-api-site-tox-checkdeletions
gate-api-site-tox-checkbuild

Now, the created jobs are ready to be run at specific events.

Gating jobs

The OpenStack CI infrastructure uses a tool called Zuul to launch jobs in response to specific gerrit events. The configuration happens via a central YAML file, it is modules/openstack_project/files/zuul/layout.yaml.
Since I like to use the same jobs for several projects, I made use of a very recent feature of Zuul called project-template. I created a template with the name "openstack-doc-gate":
project-templates:
...
  - name: openstack-doc-gate
    check:
      - gate-{name}-tox-checkniceness
      - gate-{name}-tox-checksyntax
      - gate-{name}-tox-checkdeletions
      - gate-{name}-tox-checkbuild
    gate:
      - gate-{name}-tox-checkniceness
      - gate-{name}-tox-checksyntax
      - gate-{name}-tox-checkdeletions
      - gate-{name}-tox-checkbuild

The jobs are setup to be run for each review (check) and once a patch has been approved (gate).

This template is then used in the configuration of "api-site". The name here is the name of the git repository (api-site in project openstack) and the last component (api-site) is passed as parameter to the template which results in the four jobs created in projects.yaml:
  - name: openstack/api-site
    template:
      - name: openstack-doc-gate
    post:
      - openstack-api-quick-start
      - openstack-api-site
      - openstack-api-ref

The post jobs are unchanged, these are the jobs that will be run after a patch is merged.

Additionally, I marked the checkniceness job as non-voting:
  - name: gate-api-site-tox-checkniceness
    voting: false

As mentioned in my previous post, the patch has been merged and is running just fine.
If you want to see Zuul in action, check its great status page.

Further work

I've written a followup patch to gate the API projects and operations-guide repository, it is currently up for review at https://review.openstack.org/#/c/64795/. Once this is in, the openstack-manuals can get similiar treatment. I've decided to use a separate patch for easier review since the openstack-manuals patch will remove some existing configuration.
Writting this up, I noticed that the checkniceness job is run also as non-voting gating job. This is not needed, so I wrote a simple patch to remove it: https://review.openstack.org/67702.