Feb 6, 2016

Creating new test jobs in OpenStack CI

Reviewing patches for the OpenStack CI infrastructure, there's one piece that often confuse contributors: The question how Zuul and Jenkins configuration are working together.

While we have the Infra Manual with a whole page on how to create a project - and I advise everyone to read it - , let me try to tackle the specific topic of adding new jobs from a different angle.

What we're discussing here are job, or tests, that are run. Jenkins actually runs these jobs. Zuul watches for changes in gerrit (URL for OpenStack is review.openstack.org) to trigger the appropriate jobs so that Jenkins runs them.


To understand the relationship between these two systems, let's try as an analogy programming languages: As a developer, you create a library of functions that do a variety of actions. You also write a script that uses this library to execute them. Jenkins can be considered the library of test functions. But just defining these is not enough, you have to call them. Zuul takes care of calling them, so in the analogy is your script.

So, to actually get a job running for a repository, you first need to define it in the Jenkins "library", and then you trigger its invocation in Zuul. You can also add certain conditions to limit when the job runs or whether it is voting.

If you dig deeper into Jenkins and Zuul, keep in mind that these are two different programming languages, even if both use YAML as format. Jenkins runs jobs and these are defined as text files using the Jenkins job builder. To define them, you can write a job, or use a job-template and instantiate it, or group several job-template in a job-group and instantiate that job-group to create with a few lines many jobs. Zuul uses these jobs and has as syntactic sugar templates to reuse jobs and the queues they run in.

Let's look at a simple examples, adding a new docs job to your repository called amazing-repo:

  1. Check out the project-config repository and make it ready for patch submission like creating a branch where you work on.
  2. Since for the docs job already a template exists, you can reuse it. It is called'gate-{name}-docs', so add it to your repository in file jenkins/jobs/projects.yaml:
    - project:
      name: amazing-repo
      node: bare-trusty
      jobs:
          - gate-{name}-docs

  3. Now define how to trigger the job. Edit file zuul/layout.yaml and update your repository entry to add the job:

    - name: openstack/amazing-repo
      template:
        - name: merge-check
      check:
        - gate-amazing-repo-docs
      gate:
        - gate-amazing-repo-docs

    This adds the job to both the check and gate queue. So, it will notonly be run when a patch is initially submitted for review in the check queue but also after a patch gets approved in the gate queue. Since your tree might be different when you submitted a change and when it merges, we run jobs in both situations so that the tree istested exactly as it merges.
  4. Let's go one step back: Your repository is not ready yet to havethe docs job voting, so you only want to run it as non-voting.
    In that case add a condition in the jobs section of zuul/layout.yaml:

    - name: gate-amazing-repo-docs
        voting: false


    And in your repository, only add it to the check queue. Non-voting jobs should not be in the gate, they get ignored completely and just waste resources:

      - name: openstack/amazing-repo
        template:
          - name: merge-check
        check:
          - gate-amazing-repo-docs
        gate:
          - ...

So, these are simple jobs. Stay tuned for a followup article that will cover how to use templates in Zuul - and how to modify your repository in the context of templates.

Thanks to Doug Fish for reviewing the text and giving suggestions on how to improve it - and urging me to write a follow-up.

Apr 18, 2014

OpenStack Icehouse released - and available now for openSUSE and SUSE Linux Enterprise

OpenStack Icehouse has been released and packages are available for openSUSE 13.1 and SUSE Linux Enterprise 11 SP3 in the Open Build Service.
The packages are developed in the Cloud:OpenStack:Icehouse project and can be download from the openSUSE download server:
  • For openSUSE 13.1, you can add the Icehouse repository with:
    zypper addrepo -f obs://Cloud:OpenStack:Icehouse/openSUSE_13.1 Icehouse
  • For SLES 11 SP3, add the icehouse repository with:
    zypper addrepo -f obs://Cloud:OpenStack:Icehouse/SLE_11_SP3 Icehouse
If you like to install from packages, follow the OpenStack Installation Guide for openSUSE and SUSE Linux Enterprise which has been updated for Icehouse.
With Icehouse, you can use the dashboard now in German and also use a new wizard for network creation.
Additional information about the release is available on the OpenStack web page.
My personal highlights of the new release are:
  • The translation of the dashboard to German and the new accordion navigation in the dashboard
  • The updated Installation Guides that have been greatly improved.
  • The new Database module (trove) that allows easy creation and manipulation of Databases for usage by virtual machines and thus gives you Database-as-a-Service.
  • The Compute component now allows migrations and upgrades - you can first update the controller nodes and then the compute nodes and run thus a mixture of old and new compute nodes.
  • The great progress that Orchestration makes with better integration of projects and giving the users now full control of manipulation of stacks.
  • Learning about the "most insane CI infrastructure". The more I learn about the CI infrastructure and interact with the infra team, my appreciation about their great work growth. Thanks a lot, Clark, Elizabeth, Fungi, Monty, James, Sergey, et al.!
Also, thanks to the Documentation team, it was again a lot of fun to work together with all of you and release Icehouse documentation and improve OpenStack! Team, I look forward to drink with you the 104 beers that Gauvain promised for Atlanta ;)
Now on to get Icehouse integrated into SUSE Cloud for our next release...


Apr 17, 2014

Changes for importing translations to OpenStack

Most OpenStack projects send after each commit updated files for translation to transifex. Also, every morning any translated files get merged back to the projects as a "proposal" - a normal review with all the changes in it.
Quite a lot projects had broken translation files - files with duplicate entries that transifex rejected -, and thus no update happened for some time. The problem is that these jobs run automatically and nobody gets notified when they fail.

Clark Boylan, Devananda van der Veen and myself have recently looked into this and produced several fixes to get this all working properly:
  • The broken translation files (see bug 1299349) have been fixed in time for the Icehouse release.
  • The gating has been enhanced so that no broken files can go in again.
  • The scripts that send the translations to and retrieve them from transifex have been greatly improved.
The scripts have been analyzed and a couple of problems fixed so that no more obsolete entries should show up anymore. Additionally, the proposal scripts - those that download from transifex - have been changed to not propose any files where only dates or line numbers have changed. This last change is a great optimization for the projects. For example, the sahara project got every morning a proposal that contained two line changes for each .po file - just updates of the dates. Now they do not get any update at all unless there is a real change of the translation files. A real change is either a new translation or a change of strings. For example, today's update to the operations-guide contained only a single file (see this review) since the only change was a translation update by the Czech translation team - and sahara did not get an update at all.

New User: OpenStack Proposal Bot

Previously the translation jobs where run as user "Jenkins" and now have been changed to "OpenStack Proposal Bot".

Now, another magic script showed up and welcomed the Proposal Bot to OpenStack:
Unfortunately, the workflow did not work for most projects - the bot forgot to sign the contributor license agreement:
I'm sure the infra team will find a way to have the bot sign the CLA and starting tomorrow, the import of translations will work again.

Mar 27, 2014

Horizon is translated to German for OpenStack Icehouse!

After my call for help two days ago, a few new people started helping with the German translations and we've achieved the goal of translating 100 per cent - that's all 2141 strings!
This was done by Christian Berendt, Marita Werner, Robert Simai, Sascha Peilicke, User "Schwefel", User "transistor" and myself! Thanks for getting this done!

I'm impressed that we got this final step done so quickly!

A special applause to Marita for doing the final 60 strings - I consider the last the hardest ones since those are the ones that we all skipped ;)

Mar 25, 2014

Translating OpenStack Dashboard (Horizon) - Let's get German in!

The OpenStack 18n team has recently asked for translations of the dashboard for OpenStack Icehouse, here're two emails with details (first mail, second mail).

My friend Robert suggested to do the German translations to finally have the dashboard available in German as well - and Sascha stepped up to help coordinating the German language team.

For German, there's only one resource left to translate called "openstack-dashboard-translations" - and it's available like all other OpenStack translations at transifex.


If you want to join, just do the following (the instructions apply for other languages as well, some links go directly to the German parts):
  1. Join Transifex at http://transifex.com/.
  2. Join the German OpenStack translation team and wait until your request is accepted.
  3. Go to  the German Horizon page
  4. Click on "Openstack Dashboard Translations" and start translating.

A few days ago there were 860 untranslated strings for German, now we're down to 525. Who's going to help to get it down to zero until the deadline (I've heard 4th of April)?

If you want to learn more about the OpenStack i18n team, visit this wiki page.

Jan 24, 2014

Python Virtualenv awesomeness - and developing openstack-doc-tools

Testing and developing the openstack-doc-tools package, I really came to love the python virtualenv package. The package allows installation of python packages in a local environment (virtualenv = virtual environment) for usage.

Let me guide you on how I use it for openstack-doc-tools - the repo the OpenStack documentation team uses for its tools.

First I change to the checked out openstack-doc-tools repository (get it from https://github.com/openstack/openstack-doc-tools).

Then, let's setup the virtualenv: I could setup a virtualenv manually using for example "virtualenv myenv" and install packages. But the repository knows already about virtualenv and thus I can run "python tools/install_venv.py" and it installs a virtual python environment named '.venv' using the directory '.venv' of openstack-doc-tools. The requirements for the project as mentioned in the requirements.txt and test-requirements.txt files, will get installed into this virtualenv.

Once the command finishes, it outputs a nice usage information, and I follow the option to "source .venv/bin/activate" from my shell to run commands in the virtualenv.

Next, I need to build the package via "python setup.py build" and then install the package with "python setup.py install".

Now I can easily execute commands for validating repositories. So, I can change my working directory to the api-site repository and run for example the command:
"openstack-doc-test  --api-site --check-niceness --force"

to test that the recent changes to the niceness check work fine on all files in the repository.

Jan 19, 2014

Setting up gating in the OpenStack intrastructure

One thing that impressed me with OpenStack, is the common gerrit workflow that is not only used for code but also for documentation and the infrastructure.
As I wrote in my last blog post, I enabled further validation of the documentation.
This article will describe how the validation has been done using the OpenStack CI infrastructure.

The validation is done via Jenkins and to execute a process during review, as part of validation after it is approved, after it is merged or at periodic times, you need to create jobs for Jenkins and schedule them via Zuul, the OpenStack project gating system. Let's look at an example how this can be setup.

An example of gating

Let's look at how I setup gating for the api-sites projects. The patch I sent is https://review.openstack.org/#/c/64795/ and will be used as example in this blog post.
First, you should learn about the OpenStack Continous Integration infrastructure which is documented at http://ci.openstack.org/.  Then you should check out the infra repository called config since it contains all configuration files you need to modify. Instead of directly editing Jenkins configuration, YAML files are written that the Jenkins Job Builder then uses to create Jenkins jobs.

Defining a job

For a simple job, you can just specify it. With openstack-doc-tools, I wanted to reuse the same job not only for the api-sites but for all other projects as well. This can be done with a job template and those can be grouped as job-groups. Since my jobs can be executed by tox, I could reuse the 'gate-{name}-tox-{envlist}' template (file modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml). Parameters are specified in curly braces, thus it needs as parameters the "name" of the job and the "envlist" that tox will run - basically via "tox -e envlist".
To save typing, I used job-groups to define all four jobs in one places in the file modules/openstack_project/files/jenkins_job_builder/config/manuals-jobs.yaml:
 - job-group:
    name: openstack-doc-jobs
    jobs:
      - gate-{name}-tox-{envlist}:
          envlist: checkniceness
      - gate-{name}-tox-{envlist}:
          envlist: checksyntax
      - gate-{name}-tox-{envlist}:
          envlist: checkdeletions
      - gate-{name}-tox-{envlist}:
          envlist: checkbuild

So, this creates 'gate-{name}-tox-checkniceness' etc, the variable envlist gets substituted with checkniceness for the first gate etc.

Now, to use this job-group, I added to modules/openstack_project/files/jenkins_job_builder/config/projects.yaml a project definition:
- project:
    name: api-site
    github-org: openstack
    node: precise

    jobs:
      - openstack-doc-jobs

The name parameter in the job-group is now substituted with "api-site", thus this creates the four jobs:
gate-api-site-tox-checkniceness
gate-api-site-tox-checksyntax
gate-api-site-tox-checkdeletions
gate-api-site-tox-checkbuild

Now, the created jobs are ready to be run at specific events.

Gating jobs

The OpenStack CI infrastructure uses a tool called Zuul to launch jobs in response to specific gerrit events. The configuration happens via a central YAML file, it is modules/openstack_project/files/zuul/layout.yaml.
Since I like to use the same jobs for several projects, I made use of a very recent feature of Zuul called project-template. I created a template with the name "openstack-doc-gate":
project-templates:
...
  - name: openstack-doc-gate
    check:
      - gate-{name}-tox-checkniceness
      - gate-{name}-tox-checksyntax
      - gate-{name}-tox-checkdeletions
      - gate-{name}-tox-checkbuild
    gate:
      - gate-{name}-tox-checkniceness
      - gate-{name}-tox-checksyntax
      - gate-{name}-tox-checkdeletions
      - gate-{name}-tox-checkbuild

The jobs are setup to be run for each review (check) and once a patch has been approved (gate).

This template is then used in the configuration of "api-site". The name here is the name of the git repository (api-site in project openstack) and the last component (api-site) is passed as parameter to the template which results in the four jobs created in projects.yaml:
  - name: openstack/api-site
    template:
      - name: openstack-doc-gate
    post:
      - openstack-api-quick-start
      - openstack-api-site
      - openstack-api-ref

The post jobs are unchanged, these are the jobs that will be run after a patch is merged.

Additionally, I marked the checkniceness job as non-voting:
  - name: gate-api-site-tox-checkniceness
    voting: false

As mentioned in my previous post, the patch has been merged and is running just fine.
If you want to see Zuul in action, check its great status page.

Further work

I've written a followup patch to gate the API projects and operations-guide repository, it is currently up for review at https://review.openstack.org/#/c/64795/. Once this is in, the openstack-manuals can get similiar treatment. I've decided to use a separate patch for easier review since the openstack-manuals patch will remove some existing configuration.
Writting this up, I noticed that the checkniceness job is run also as non-voting gating job. This is not needed, so I wrote a simple patch to remove it: https://review.openstack.org/67702.