Aug 12, 2016

Document Binary Package Dependencies - not only for OpenStack Python Packages

Python developers record their dependencies on other Python packages in requirements.txt and test-requirements.txt. But some packages havedependencies outside of python and we should document thesedependencies as well so that operators, developers, and CI systems
know what needs to be available for their programs.

Bindep is a solution to this, it allows a repo to document binarydependencies in a single file. It even enablies specification of which distribution the package belongs to - Debian, Fedora, Gentoo, openSUSE, RHEL, SLES and Ubuntu have different package names - and allows profiles, like a test profile.

Bindep is one of the tools the OpenStack Infrastructure team has written and maintains. It is in use by already over 130 repositories.

For better bindep adoption, in the just released bindep 2.1.0 we have changed the name of the default file used by bindep from other-requirements.txt to bindep.txt and have pushed changes to master branches of repositories for this.

Projects are encouraged to create their own bindep files. Besides documenting what is required, it also gives a speedup in running tests since you install only what you need and not all packages that some other project might need and are installed  by default. Each test system comes with a basic installation and then we either add the repo defined package list or the large default list.

In the OpenStack CI infrastructure, we use the "test" profile for installation of packages. This allows projects to document their run time dependencies - the default packages - and the additional packages needed for testing.

Be aware that bindep is not used by devstack based tests, those have their own way to document dependencies.

A side effect is that your tests run faster, since they have less packages to install. A Ubuntu Xenial test node installs 140 packages and that can take between 2 and 5 minutes. With a smaller bindep file, this can change.

Let's look at the log file for a normal installation with using the default dependencies:
2 upgraded, 139 newly installed, 0 to remove and 41 not upgraded
Need to get 148 MB of archives.
After this operation, 665 MB of additional disk space will be used.

Compare this with the openstack-manuals repostiry that uses bindep - this example was 20 seconds and not minutes:
0 upgraded, 17 newly installed, 0 to remove and 43 not upgraded.
Need to get 35.8 MB of archives.
After this operation, 128 MB of additional disk space will be used.

If you want to learn more about bindep, read the Infra Manual on package requirements 
 
If you have questions about bindep, feel free to ask the Infra team on #openstack-infra.
 
Thanks to Anita for reviewing and improving this blog post and to the OpenStack Infra team that maintains bindep, especially to Jeremy Stanley and Robert Collins.

Aug 11, 2016

Testing OpenStack with always updating python package versions

With any software package, you will need additional packages to run it. Often, there's a tight coupling: The software package will only run with specific other package versions. This dependency information is sometimes found in README files, in code, or in package metadata. If you install the package, you need to figure out the dependency and
handle it properly.

The Python package installer pip uses a list of requirements to install dependent Python packages. This list not only contains the name of packages but also limits which versions to use, or not to use.
In OpenStack we handle these dependencies in a global requirements list and use it for most of the repositories. During initial testing a specific package version is tested but at a later point, another one might be used, or during deployment again another one.

To document what was tested, give guidenance for deployment, and help to figure out breakage by upstream projects, the OpenStack requirements projects maintains a set of constraints with packages pinned to specific package versions that are known to be working.
These are in the upper-constraints.txt file.

Devstack already handles upper-constraints.txt when installing packages and I'm happy to say that tox, the Python testing framework used in OpenStack, can now handle upper-constraints as well everywhere.


Constraints for tox based jobs

To use constraints, change in tox.ini the install command to:

install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}

Caveat

Note that constraints are used for the installation of each packages, so if you want to install a package from source and have constraints for a specific version in the constraints file, it will not work. This happens with some of  the OpenStack python client packages: When they install their dependencies, those might
have a dependency on the client package itself. And this then will cause an error since the client package should get installed from source.

So, projects need to remove the constraints file for themselves if they run into this. Packages like python-novaclient and python-glanceclient therefore use a wrapper (tools/tox_install.sh) as
install command to edit the constraints file first and remove their own project from it.

Also, be aware that this only for those jobs that have been enabled for it in the project-config repository. It's done for all the generic tox enabled targets and should be done for all custom tox targets as well. Some repositories are not using constraints like project-config
itself, so those jobs are not set up.

Constraints for DevStack jobs

Devstack-gate takes care using constraints, there is nothing for a repository to do to honor constraints.

Check the devstacklog.txt file, if constraints are in use it will use lines like:

Collecting oslo.context===2.7.0 (from -c /opt/stack/new/requirements/upper-constraints.txt (line 204))

References

To learn more about constraints read the requirements documents. There is also a spec that explains all the steps that where needed for this.


Thanks

As usual in OpenStack, such work is a team work of many people. I'd like to thank especially:

  • Robert Collins 'lifeless': For writing the initial spec, implementation work, and giving guideance on many of these details.
  • Sean Dague: He was bold enough to try using constraints everywhere and showing us where it failed.
  • Sachi King for making zuul-cloner usable in the post queue. This was a missing part in the last months.
  • The OpenStack infra team for many reviews and design discussions - especially to Jeremy Stanley and Jim Blair.