AJ's Open Source, openSUSE and SUSE Ramblings

Chef and Crowbar for setting up OpenStack clouds - with openSUSE and SUSE Cloud

2013-04-12T12:16:00.000+02:00

To easily setup an OpenStack-based cloud on server hardware, we at SUSE use the open source Crowbar project. Crowbar eases the setup of all the different OpenStack components and thus reduces the time spent deploying and managing an OpenStack-based cloud.

Crowbar is a platform for server provisioning and deployment from bare metal. It provides server discovery, firmware upgrades, and operating system installation using PXE Boot, and it deploys applications on top of functioning operating systems using Chef.

Currently development is happening on Crowbar 2 and I'd like to give a short overview of a few things that are done here to make this work useable for openSUSE and SUSE Cloud - SUSE's OpenStack based cloud offering.

Moving to a common Chef cookbase base

So far, Crowbar comes with its own Chef cookbooks to setup OpenStack, a forked and heavily adapted version.

We'd like to have one common set of Chef cookbooks that are shared between Crowbar and administrators using Chef directly. As Rob Hirschfeld said: "The real victory comes when multiple deployments use the same trunk instead of forking".
The additional benefit here is that these cookbooks will then also work without Crowbar on e.g. openSUSE and SUSE Linux Enterprise.

Right now it looks like the awesome cookbooks done by AT&T and Rackspace (e.g. https://github.com/att-cloud/cookbook-keystone/blob/master/README.md) are the best option for going forward. But these cookbooks have two limitations for our usage: they do not handle openSUSE and SUSE Linux Enterprise and are not ported for Grizzly yet. So, this gives us the next tasks:

Enhance existing Chef cookbooks to know about openSUSE and SUSE - and for Grizzly

We're enhancing these cookbooks now so that they support OpenStack Grizzly and we're also adding support for servers running openSUSE and SUSE Linux Enterprise.

Once the cookbooks are usuable on their own, they can be integrated into Crowbar.

Integrate the Chef cookbooks into Crowbar

To use the cookbooks from Crowbar, Crowbar needs to pass parameters to
them. This concept is called "Attribute Injection" and Judd Maltin has explained this in detail in a great blog post.

Crowbar 2 and Chef / Puppet

Of interest to some might be that in the coming Crowbar 2, the Chef usage has been hidden behind an extra layer called "Jig". Jig was introduced to overcome some design limitations of Crowbar, the extra abstraction also gives a more flexible framework. Now other configuration management options can be added besides Chef. Puppet is one framework that the Crowbar team would like to support as backend and this work is currently postponed after the initial Crowbar 2 release.

OpenStack Grizzly Packages for openSUSE and SUSE Linux Enterprise

2013-04-09T09:02:00.000+02:00

Last week the OpenStack project released its 7th release named Grizzly and the packages are now available for openSUSE and SUSE Linux Enterprise via the openSUSE Build Service. Packages can be downloaded from the download server.

The repositories contain all the core OpenStack packages plus incubated projects that will be part of core in the Havana release (Heat for VM orchestration and Ceilometer for metering). Additional package dependencies are included as well.

For discussion about these packages, use the opensuse-cloud at opensuse.org mailing list and the freenode channel #opensuse-cloud. To learn more about OpenStack on openSUSE, see also our wiki page.

Moving to open development: OpenStack / Crowbar / Chef on SUSE

2013-03-11T12:00:00.000+01:00

This blog post was written by Adam Spiers and Andreas Jaeger.
Here at SUSE we have based our business on Free and Open Source Software for over 20 years, so it is nothing new to say that we strongly believe in open development and collaboration as a way of producing software of the highest quality.
Therefore we are pleased to announce that we have opened up our work on development and packaging of OpenStack, Crowbar, and Chef, for both openSUSE and SUSE Linux Enterprise. This work is happening publically in the Open Build Service, and we warmly invite everyone to use the results and also join in with development, testing, documentation and packaging.
Open development is a process that does not end with code on public servers, but involves ongoing open communication as well. Please speak up on the opensuse-cloud mailing listif you see any areas where we could improve further!

OpenStack

OpenStack is our preferred cloud management platform, due to its open nature and backing from an already huge and rapidly growing community. We have packaged both OpenStack's Folsom releaseand the beta packages for the upcoming Grizzly release and have made them available in the Open Build Service.

Crowbar

OpenStack's incredible flexibility can also make it difficult to deploy. Therefore to ease the set-up of all the different OpenStack components out of the box, we are supporting the open source Crowbar project. Our partners and customers find that using Crowbar dramatically reduces the time spent deploying and managing an OpenStack-based cloud.
Crowbar is a platform for server provisioning and deployment from bare metal. It provides server discovery, firmware upgrades, and operating system installation using PXE Boot, and it deploys applications on top of functioning operating systems using Chef.

Opening up Crowbar development

Similarly to how OpenStack was born at RackSpace and later grew into an independent entity in the form of the OpenStack Foundation, Crowbar was originally developed by Dell engineers, and is now a fully-fledged Open Source project involving close collaboration with SUSE and others. There are weekly public meetings, a public mailing list, a #crowbar IRC channel, public Trello boards and so on. As an indication of the project's independent nature, the authoritative location for the git repositories has changed from https://github.com/dellcloudedge to https://github.com/crowbar, and a new homepageis currently under construction.

Crowbar 2.0

Work on the upcoming 2.0 release of Crowbar is proceeding at a furious pace, and packages for openSUSE and SUSE Linux Enterprise are available from the OBS. You are very welcome to join in and find out more!

Chef packages

Chef is an Open Source configuration management tool that allows remote administration of systems at scale. It has a thriving communitywhich has a large overlap with the OpenStack community. We have packaged both Chef 10and Chef 11for openSUSE and SUSE Linux Enterprise.

Tying it all together

Since each major release of these projects is packaged separately in the Open Build Service, you are free to "mix'n'match" which of these components you want to use together:

OpenStack Essex
OpenStack Folsom
OpenStack Grizzly
Crowbar 2.0
Chef 10
Chef 11

Of course, each combination will work differently. To try any of them out, simply navigate to one of the project's Repository tabto obtain the link to the relevant download repository. Then you can add that to your host's list of software repositories in the normal way via YaST2 or zypper, and start installing packages from it. Alternatively, you can install directly via one-click install.

Automation and Continuous Integration

We use Jenkins to automatically package changes from upstream git repositories into rpms within an openSUSE environment, and if they successfully build and install, Jenkins commits those changes to the Open Build Service (OBS) which then automatically builds and publishes packages and images. Jenkins also does unit tests as well as full stack tests. By automating the packaging and testing process, we can spend more time on development while maintaining high quality, and gain more clarity around where problems exist.
You can also read in more detail about our development model using the Open Build Service.

Finding out more

If you have questions, ask them on the opensuse-cloud mailing list or on the recently launched #opensuse-cloud freenode IRC channel.
You can also find more details via the openSUSE wiki:

This is work in progress, so feel free to help improve anything!

Coordinating efforts to create OpenStack packages for openSUSE and SUSE Linux Enterprise Server

2013-02-25T14:41:00.000+01:00

Engineers from B1 Systems and SUSE have now joined forces to package OpenStack in the Open Build Service for openSUSE and SUSE Linux Enterprise. The B1 Systems engineers bring in their OpenStack consulting and packaging expertise, the SUSE team their packaging and hardening expertise from building a commercial OpenStack based release with SUSE Cloud.

The work is done in the Open Build Service which allows building of packages for various distributions and versions of it. There are stable packages available from the latest OpenStack release, Folsom, and beta packages of the upcoming Grizzly release. These OpenStack packages are build now for openSUSE 12.2, openSUSE 12.3 and SUSE Linux Enterprise 11 SP2.
The joint development project is contained in the Cloud:OpenStack and its subprojects.

Additionally, OpenStack Folsom is part of the upcoming openSUSE 12.3 release. Thus, you can build, administrate and use an OpenStack cloud within openSUSE.

If you like to use OpenStack packages or join the team, please join us on the mailing list opensuse-cloud@opensuse.org (subscribing information is here).

I'll talk in future posts on the development progress of these packages in the Open Build Service. In the meantime, to get more information about OpenStack on openSUSE, see the openSUSE OpenStack Portal.

Some background: B1 Systems is a consulting partner of SUSE and offers support for OpenStack on openSUSE and SUSE Linux Enterprise. Some of their engineers have been packaging OpenStack for over two years now (starting with Bexar) and made those packages available through the Open Build Service for both openSUSE and SUSE Linux Enterprise. SUSE has delivered last year its SUSE Cloud 1.0 release that is based on OpenStack Essex.

Easy way to install TeX packages for openSUSE 12.3

2012-12-21T11:21:00.004+01:00

With the upcoming openSUSE 12.3, the TeX installation is much more modular - but contains now all of TeXlive. Werner added a nice trick to install packages.
If you get an error like:
LaTeX Error: File `multirow.sty' not found.
You can now easily install the missing package with just saying "zypper in 'tex(multirow.sty)'".
Similar, if a font file is missing, e.g. you get a message like this:

! Font U/pzd/m/n/10=pzdr at 10.0pt not loadable: Metric (TFM) file not found.you can install the file with "zypper in 'tex(pzdr.tfm)'".

So, with the new capability 'tex', you can easily install missing files.

Glibc 2.17 on the finishing line

2012-11-20T21:57:00.000+01:00

In the next few days David Miller as GNU C Library (glibc) 2.17 release manager will call a freeze for glibc development and I've packaged glibc for openSUSE and compiled the whole distribution with it - and it looks fine.
For those curious, what's in the new glibc, here's an incomplete list of the major changes. For the full list, see ChangeLog file and the NEWS file. The list comes from the current NEWS file and I've only copied entries relevant to openSUSE:

Port to ARM AArch64 contributed by Linaro.
The new function secure_getenv allows secure access to the environment, returning NULL if running in a SUID/SGID process. This function replaces the internal function __secure_getenv.
SystemTap static probes have been added into the dynamic linker. Implemented by Gary Benson.
Optimizations of string functions strstr, strcasestr and memmem. Implemented by Maxim Kuvyrkov.
The ttyname and ttyname_r functions on Linux now fall back to searching for the tty file descriptor in /dev/pts or /dev if /proc is not available. This allows creation of chroots without the procfs mounted on /proc.
The `crypt' function now fails if passed salt bytes that violate the specification for those values. On Linux, the `crypt' function will consult /proc/sys/crypto/fips_enabled to determine if "FIPS mode" is enabled, and fail on encrypted strings using the MD5 or DES algorithm when the mode is enabled.
The `clock_*' suite of functions (declared in <time.h>) is now available directly in the main C library. Previously it was necessary to link with -lrt to use these functions. This change has the effect that a single-threaded program that uses a function such as `clock_gettime' (and is not linked with -lrt) will no longer implicitly load the pthreads library at runtime and so will not suffer the overheads associated with multi-thread support in other code such as the C++ runtime library.

Also, 125 bugreports have been marked as fixed, a couple of changes from eglibc got merged (especially for cross-testing) and some cleanups of the code were done.

With the current git development version packaged as rpm, the openSUSE Build Service compiled the whole distribution on x86-64. There were three different kind of problems in packages which showed up as build errors:

Some packages had missing includes (e.g. signal.h or stdint.h). Those are easily fixed by including the header defining the missing types.
The functions setfsgid() and setfsuid() produce warnings when the return value is not checked and thus fail to when -Werror is used. The proper fix is to check whether these functions have a return value less than 0.
clock_gettime() was moved from librt to libc. Some configure scripts check only whether clock_gettime is in librt and assume that other functions like mq_gettattr() are in the same library. So, the configure check for clock_gettime() in librt needs to be extended to look for other functions as well.

I did not noticed directly any problems in glibc itself, just these three kind of problems in packages - and had to fix around 10 packages at all.
This was all done with the git version from the 9th of November, I've updated now to the current git version and will retest.
My plan is to push the current glibc package soon to the openSUSE development tree called openSUSE Factory.

Updates to openSUSE Packaging Guidelines

2012-11-07T10:10:00.000+01:00

On the opensuse-packaging mailing list, we've recently formed a team that will take care of the packaging guidelines and introduced a small process to change them.

As part of that process, we're announcing regularly the changes to the packaging guidelines. Since this is a first such announcement, it is not a complete change but just points out a few things from the past few months. In the future, we will send out this email once a month.

The Packaging guidelines can be found in the openSUSE wiki at http://en.opensuse.org/openSUSE:Packaging_guidelines.

The list of changes that I'd like to mention are:

New Lua Guidelines
Reworked font guidelines
Documenting changes in packages
Teams involved

New Lua Guidelines

We now have guidelines for lua at http://en.opensuse.org/openSUSE:Packaging_Lua.

Reworked font guidelines

The packaging of fonts has been completely changed, and is documented at http://en.opensuse.org/openSUSE:Packaging_Fonts.

Documenting changes in packages

The openSUSE review team is now also enforcing proper documenting
changes in packages:

First, the .changes entry (rpm changelog) surves two purposes:

News for the user
History tracking of packaging changes (often referenced in bugs to verify if a user has the latest packaging bugfixes).

Information about updates

A simple "Update to version x.y.z" is, as before, not accepted. There should be some buzz around the update for the user; some major reasons to the upgrade should be listed.

Changes on the package itself should be mentioned in a way that any other contributor to the same package can follow traces of why something is the way it is. Commonly, added (build)dependencies are interesting to be seen, special hacks to make something work in a particular way [..]: Always consider that package maintenance is a distributed task and various contributors need to be able to step up at will.

Documenting patch life cycle

The rules about patches are listed at http://en.opensuse.org/openSUSE:Packaging_Patches_guidelines .

Most prominent is likely the mentioning of the patches life cycle, which forces you to mention additions and removals of patches in the changelog. As history shows, this can be helpful if a patch got removed, and later a regression is reported; finding out when a patch was removed can be crucial in reconstructing feature sets (including contacting the contributor that dropped it.. which is easily extracted from the .changes if listed).

The main appeal is to the devel project maintainers / reviewers, to keep out for those rules, to live according to them, as it is frustrating for everybody if a package needs to be declined by the openSUSE Factory Review team:

The dev prj maintainer is the one getting the 'decline' (as it was usually a forwarded request), which often leaves the 'fixing' to the devel project maintainers, where the 'originator' of the fix would have been willing to actually do that...

Note: The review team is not enforcing the usage of patch markup unless the package already follows this convention.

Teams involved, contact

I mentioned two teams previously, these are the openSUSE review team
and the team taking care of the guidelines. You can reach both via the opensuse-packaging mailing list.

Changing my responsibilities @ SUSE

2012-11-06T09:15:00.000+01:00

As you already know, during the last few months, SUSE has reorganized responsibilities and resources around openSUSE. As a result of this new strategy, the openSUSE Team has been created (with Agustin as team lead), formed by those SUSE employees who work full time in the community project.

I've worked with the openSUSE team to hand over my openSUSE related responsibilities to them and now it's time for me to formally take over some new responsibilities inside SUSE:

I'll continue to be part of the SUSE Product Management team - now as product manager - and continue driving SUSE Linux Enterprise 12 efforts with the other product managers and engineering.
openSUSE is the upstream of SUSE Linux Enterprise and with my help driving development and keeping SUSE a good partner in the openSUSE ecosystem, I'll continue to be involved. Also, I will continue to contribute to openSUSE with packaging and reviewing of packages.

As SUSE product manager I'm also joining the efforts around cloud, e.g. SUSE Cloud, and SUSE Studio and will also take product management ownership for the toolchain (compiler, debugger, C library, ...).

openSUSE Summit 2012

2012-09-24T01:03:00.000+02:00

Friday afternoon the openSUSE summit started. Registration opened in the morning and thanks to Kenneth, Vincent and Alan for welcoming everybody!

The afternoon had a few sessions and then the openSUSE Summit Opening Ceremony started. opened the event and then Andy Wafaa and Michael Miller talked about the openSUSE project - and gave out large geekos to Vincent as new openSUSE chairperson, Alan Clark as passing person, Izabel Valverde for her work on the travel sponsorship team and to Bryen Yunashko for his work as co-chair.

Andy and Michael were also distributing openSUSE 12.2 DVDs and signed them. So, I'm a proud owner of a special 12.2 DVD!

Our pool party in the evening was moved inside due to rain. Alan had bought many small battery-operated boats that were build together and then they had to race over the pool. Many of them took a few detours and went in circles; some tried to dive - so a search and rescue team was launched to rescue some of the boats. It was a great fun.

Saturday morning Stefano Maffulli from the openstack foundation talked about open source and the cloud in his keynote.

Afterwords many sessions were held. As part of the openSUSE Summit, some GNOME developers met for a hackfest and Bryen organized a raffle to help sponsoring them.

In the evening we had a Hawaii party with Hawaii music and dancing.

Sunday started again with sessions and then Bryen handed out certificates of thanks to everybody involved in the success of the event. Then we had a big discussion round (called "general session") about what we liked and disliked at the conference and about openSUSE in general. There was also a discussion on the role of the board - should they be passive or actively doing things? It was suggested that the board actively pushes team to move forward and helps them do what they need to be successful.

It was great to talk with many people - many I've never met before. The geeko lounge was a great place to relax and discuss.
Thanks for a great event especially to the co-chairs Bryen Yunashko and Alan Clark, to the SUSEcon organization team for all their organization, to our sponsors SUSE, owncloud and omnibond.
I've uploaded a few more photos from the summit to an album.

SUSECon 2012

2012-09-22T14:18:00.005+02:00

On Monday I flew to Orlando, Florida, to attend SUSECon 2012 - the "first annual SUSE conference". Orlando greeted us with some rain clouds and a high humidity. So far, every morning starts nice and warm and later on we got out share of rain.

Tuesday was building up of the show and I made familiar with my booth and a final go through my presentations. On Tuesday evening, the opening reception was held and on Wednesday the event was opened with a keynote by SUSE's Nils Brauckmann and Michael Miller, IBM's Doug Balog and Intel's Dirk Hohndel (see the SUSE youtube channel for some videos of the keynotes).

I gave my presentation about "openSUSE - upstream of SUSE Linux Enterprise" first on Wednesday and repeated it on Thursday and was attending a booth and talking with people in the exhibition hall.

Wednesday evening the big "20 years SUSE Birthday Party" was held in the exhibition floor and the video of SUSE Gangnam style shows what fun we had:

Thursday's evening activity included the Blue Man group - a really great performance. We also had some fun coming back from the event through the rain but thanks to great organization we all got ponchos.

>On Friday morning I was part of the "SUSE Linux Enterprise 12" roundtable presentation and then it was already time for the closing keynote by Michael Miller and Dell's John Igoe. The topic of the keynote was Cloud - and what changes it will bring to the IT.

SUSECon was great - it was great meeting customers, colleagues and openSUSE community incl the openSUSE Forums team.
Now it's time for the openSUSE Summit.
I've uploaded some more photos to the SUSECon event page.

UEFI Secure Boot and openSUSE

2012-08-10T20:43:00.000+02:00

UEFI Secure Boot is a feature that is coming soon with new PCs. Traditional BIOS booting is getting replaced now with UEFI and now hardware vendors are going - following requirements from Windows 8 - to use UEFI for booting and enable UEFI Secure Boot. This cases a number of problems and challenges for Open Source users.

My colleagues Vojtech Pavlik and Olaf Kirch have been looking into what UEFI Secure Boot is, what their effect on SUSE and openSUSE is and they propose a solution to handle it.
They have written three blog posts explaing SUSE's approach to secure boot:

Matthew Garret from Red Hat is driving the Fedora implementation of Secure Boot and he blogged as well about the solution and called it "elegant" - and expects to use it for Fedora as well.

Thorsten Leemhuis has summarized this in a nice article at the H (German original at Heise).

The openSUSE community is now discussing whether we can use the SUSE proposal also for openSUSE and the first reactions seems that this is the way to go.

There's not much sense in summarizing the articles further, Thorsten did a great job, so read his article - or read the full three posts plus Matthew's comment for all the details.

Glibc 2.16 for openSUSE Factory

2012-08-03T12:52:00.002+02:00

The GNU C Library (glibc 2.16) has been released a couple of weeks ago and is the first release after the changed glibc development model. It looks like the release with the most bug fixes and most contributors.
The most significant developer visible changes are IMO:

Support for ISO C11
Many bug fixes for the math library thanks to a thorough audit of all open math library bugs. The results are mor accurate exceptions and function results.
Support for the x32 ABI on x86-64 which also brings unified headers for x86 and x86-64 - so there are really the same headers for all three ABIs on x86-64.

I've packaged glibc 2.16 in the openSUSE Build Service and could reduce the number of patches we had from 61 to 39 - since many of our patches were upstreamed during 2.16 development and a few backported from it. I also classified the patches to mark which ones are upstreamed and which ones are not appropriate for upstream (yet).

While the focus of openSUSE is on getting 12.2 out of the door, I worked on getting glibc 2.16 ready for the next release in a side project. The library itself builds fine and Coolo has setup a staging project called openSUSE:Factory:Staging:Glibc where the whole distribution is build with the new glibc (in this case we use 12.2 as a base since that one is moving slower).

Fixing Packages

I've had to fix a small number of packages to get them building with glibc 2.16 and pushed everything directly to factory.

The issues I encountered and fixed where:

Missing <sys/resource.h>

The include files have been updated and some programs did not have an include on <sys/resource.h> and instead include some other file that included it.

The build failure looks like:

[   89s] util_exec.cpp: In function 'void dmtcp::Util::adjustRlimitStack()':
[   89s] util_exec.cpp:374:19: error: aggregate 'dmtcp::Util::adjustRlimitStack()::rlimit rlim' has incomplete type and cannot be defined
[   89s] util_exec.cpp:375:15: error: 'RLIMIT_STACK' was not declared in this scope
[   89s] util_exec.cpp:375:34: error: 'getrlimit' was not declared in this scope
[   89s] util_exec.cpp:376:26: error: 'RLIM_INFINITY' was not declared in this scope
[   89s] util_exec.cpp:384:36: error: 'setrlimit' was not declared in this scope

No declaration anymore for gets

gets is not declared anymore as part of ISO C11 and therefore code using gets needs to be compiled using older standards - or better do not build with gets.
The packages that failed where packages using gnulib that issued a warning when gets was used - and this warning code works only if gets is declared. Once these packages update their gnulib code, everything is fine, for now the warning could just be disabled or only enabled when there is a declaration.

No more 'struct siginfo'

glibc 2.16 removes the undocumented definition of 'struct siginfo'
from <bits/siginfo.h>. This struct was always a typedef for 'siginfo_t' which is required by POSIX, so 'siginfo_t' should be used instead.

TIME_UTC from ISO C11 in boost

ISO C11 introduced the constant TIME_UTC - and boost also declares this value. This breaks building with boost and thus upstream boost now renamed its value to 'TIME_UTC_' which is an API change.
So, boost and all packages using boost's boost::TIME_UTC needed fixing.

_FORTIFY_SOURCE requires compilation with optimization

Building with -D_FORTIFY_SOURCE is done by default for openSUSE but the extra features are defined in a way that they only work with optimization. Now glibc complains if a file is compiled without optimization but with this feature macro.
There were two packages that contained -O0 for a few files that were easily fixed.

Fixing packages in the Build Service

To build locally a package with the new glibc, just issue the following osc command:

osc build --alternative-project=openSUSE:Factory:Staging:Glibc standard x86_64

Next step: Factory

Once all packages are building fine (so far I had to fix 50+ packages, the other 5000+ are fine), I'll discuss with Coolo when and how to submit this to Factory.

GCC 4.7 C++ ABI changes and openSUSE 12.2

2012-07-20T15:47:00.000+02:00

GCC 4.7 introduced two changes that resulted in different ABIs (Application Binary Interface) for files that were compiled with C++98 versus files that were compiled using C++11 options. This could result in crashing applications, like bug bnc#767666 where "a C++98 library picked up a std::list operator= from a c++11 library resulting in a crash."

openSUSE 12.2 has been hit some of these incompatibilities in the update stack and the GCC developers now updated the compiler collection for openSUSE 12.2 to the current GCC 4.7 branch head which includes fixes for the two known issues. Those changes will also be in the upcoming GCC 4.7.2 release.

Every binary that uses std::list or std::pair in C++11 mode was effected and Coolo decided to rebuild the complete openSUSE 12.2 distribution to catch really everything - besides the known problems with the update stack and LibreOffice. This rebuild has been done after RC1 and therefore if you update from openSUSE 12.2 RC1 to RC2, you will see updates for many packages since they were recompiled.

This is also a warning - if you're using GCC 4.7.0 or 4.7.1 without this fix and compile programs using C++11 mode, update your compiler and recompile.

Some background

By default GCC compiles C++ programs using the options "-std=gnu++98" which is the GNU dialect of C++98. To build for the new C++11 standard, you have to specify "-std=c++11".

Details from the GCC changes pagel:

"GCC versions 4.7.0 and 4.7.1 had changes to the C++ standard library which affected the ABI in C++11 mode: a data member was added to std::list changing its size and altering the definitions of some member functions, and std::pair's move constructor was non-trivial which altered the calling convention for functions with std::pair arguments or return types. The ABI incompatibilities have been fixed for GCC version 4.7.2 but as a result C++11 code compiled with GCC 4.7.0 or 4.7.1 may be incompatible with C++11 code compiled with different GCC versions and with C++98/C++03 code compiled with any version."

Thanks

The fixing included many GCC jdevelopers, I'd like to give a special thank you to Richard Günther und Michael Matz for debugging the issues in openSUSE and helping to get them fixed both upstream for GCC 4.7.2 and in openSUSE's GCC.

openSUSE 12.2 vs Factory Development

2012-07-13T13:37:00.001+02:00

On the factory mailing list there has been some discussion about testing for openSUSE 12.2, so let me try to summarize the situation again: Coolo has split off the 12.2 distribution and its building on its own in the openSUSE Build Service as project openSUSE:12.2. Factory is its own project as usual.

Check-in

There are two ways to get packages into openSUSE 12.2 right now: The one way is to create a maintenance request against openSUSE 12.2 - the same way it's done for released distributions. The other way is a submission to factory, the release team reviews all submissions for Factory and decides which ones to take for 12.2 as well. So, if you really want your change in 12.2, you better state it as part of the submit message and write an entry for changes that makes the release team want the change.

Publishing

The openSUSE maintenance team likes to get everything - but the packages that went in via maintenancerequests as well as the packages that went in via Factory - out as updates. So, if you have the 12.2 and the 12.2-non-oss repositorires (from http://download.opensuse.org/update/ as repositories, you'll stay up to date.
Once RC2 is published, a full tree gets published together with new ISOs.
Note, that we now split the update repository to have a separate update repository for packages that are not Open Source.
Update 2012-07-14: During an installation of openSUSE 12.2 RC1, you get already the path of the 12.2 repositories. If you don't have them, enter directly as repository http://download.opensuse.org/distribution/12.2/repo/oss/ and http://download.opensuse.org/distribution/12.2/repo/non-oss/. The links are redirects right now and point to RC1 - and will be updated with future releases.

Factory development

So, this also means that Factory is open for intrusive changes. I ask everybody to concentrate on making a great openSUSE 12.2 but the work for 12.3 can go in now.

Interactive Package Review

2012-06-29T10:56:00.000+02:00

I've reviewed a couple of dozen packages this week for openSUSE:Factory (Coolo checked some of these into 12.2 as well) in the openSUSE Build Service and Darix showed me a nice way to do the review: interactive review with osc.

Note: This depends on the just released osc version 0.135 (get it from the openSUSE:Tools repository).

So, open up a large terminal window, enter "osc request list -U <your-id> -i" and go through the list. It will show you the submit request and offer you the following options (the letter in (brackets) is the letter to type, e.g. i for diff):
"d(i)ff/(a)ccept/(d)ecline/(r)evoke/(b)uildstatus/c(l)one/(e)dit/(s)kip/(c)ancel"

The important options are:

diff: Opens your editor with a diff of the changes
accept: Accept the review/request (you can give some commend with "-m comment")
decline: Decline the review/request. If you requested the diff before, it will open the diff again in your editor and allow you to easily copy and paste some lines to your decline message which will be added at the top of the diff.
revoke: Revoke the request
buildstatus: Show whether the package has been build.
skip: Ignore the review/request for now, go to the next one.
cancel: Stop reviewing.

Some examples:

osc review list -G autobuild-team openSUSE:Factory -i:
Review interactively all packages that has an open review for group "autobuild-team" and as target project openSUSE:Factory.
osc request list -U a_jaeger -i:
Review interactively all packages that have an open submit request and that I'm (my user is a_jaeger in the openSUSE Build Service) allowed to approve/decline.

If you're part of a group, you should add to your ~/.oscrc also the option "review_inherit_group = 1 ". This way, it will not ask you for which group you do the review but take the one you used as command line ("-G autobuild-team").

Full TeX Live modularized for openSUSE 12.2 - 6000+ packages coming

2012-05-24T14:23:00.003+02:00

Werner Fink - the SUSE TeX guru, master of bash and scripting - really surprised me today. He just increased the number of binary RPMs in a significant way with adding 6000 packages from the full TeX Live distribution. So, if we can convince him to add 3000 more packages, we would have 30000 rpm packages.

So, what has he done? He has downloaded the TeX Live package database, applies his self-written perl parser to it to generate 2200 spec files out of it (including recording the dependencies that the package database contains) and additionally generates some collection packages and schemes - and also builds the binary packages. The split in these packages, collections and schemes is part of TeX Live, Werner put them with his generator in rpms.
Collection packages just require other packages and are topic related, e.g. there's a collection package called texlive-collection-latex, if you install it, it installs everything you need for typesetting LaTeX files.
Schemes are the basis for everything, there's for example a small (texlive-scheme-small) and a full scheme (texlive-scheme-full).
So, you could install the small scheme and enhance it with the LaTeX and music collections.

For building the 2200 packages (they contain fonts, documentation, TeX code), Werner wrote a meta package that runs rpmbuild in a loop for those packages. So, the build service sees only one package (which results in 6000 rpm packages) and not 2200.

So, TeX on openSUSE 12.2 will have 83 collections and 9 schemes, a few source packages, 6000 rpm packages - for a total of 1.4 GB plus 52 MB for the binaries of TeX. Fortunately the 6000 rpm packages are of architecture "noarch", so we only have them once in the openSUSE repository.
Previous openSUSE releases only contained the normal TeX Live distribution with a small number of very large packages, now openSUSE contains everything in a modular way.

All of this is available in the openSUSE Build Service as part of the TeX Live project and soon in openSUSE 12.2.

Security or Convenience? Defining a better policy

2012-05-22T14:46:00.000+02:00

The openSUSE security concepts have been changed gradually over the years with new tools like PolicyKit, PolKit and its usage in system tools.

It's time now to step back, and review what we have and want.

Marcus and Ludwig from the SUSE security team and myself have discussed over the last weeks a bit and like to open this to a broader round now to get your help defining what needs to be done.

Challenges we face

Administrating a system in a secure way is always balancing the needs and requests of security, convenience and usability. There's also the additional challenge that upstream projects often have a different view on either of these and therefore make different decisions and influencing upstream projects is quite often a difficult task.

Background

Linus Torvalds in his Google+ rant said:

"I first spent weeks arguing on a bugzilla that the security policy of
requiring the root password for changing the timezone and adding a new
wireless network was moronic and wrong.

I think the wireless network thing finally did get fixed, but the
timezone never did - it still asks for the admin password.

And today Daniela calls me from school, because she can't add the
school printer without the admin password.

...
So here's a plea: if you have anything to do with security in a
distro, and think that my kids (replace "my kids" with "sales people
on the road" if you think your main customers are businesses) need to
have the root password to access some wireless network, or to be able
to print out a paper, or to change the date-and-time settings, ..."

How to continue?

We've collected a couple of use cases for the administration of a
local system at: http://en.opensuse.org/openSUSE:Security_use_cases

For each use case we added a short security evaluation but in most cases don't give a recommendation on what to do.

I now have a call for action: Review and discuss the contents of the wiki page
using the following questions:

Are there any use cases missing?
Is there any thing missing in the specific use cases?
How can we solve these use cases so that a system is easy to setup for the most common usage scenarios?

Let's do the discussion on the opensuse-factory mailing list, I'll update the document with any improvements. Feel free to enhance it as well.

openSUSE - the upstream of SUSE Linux Enterprise

2012-05-07T15:47:00.000+02:00

openSUSE contains many independent software projects that have an "upstream" - for example the GNOME desktop in openSUSE is developed mainly by the upstream GNOME community, and the openSUSE developers add integration to make a distribution polished and easy to use. Integration consists of packaging, testing, bugfixing and some extra customization and development. Also the openSUSE developers talk with the GNOME developers about their needs and problems.

It's good practice to push everything upstream and then the upstream community might just accept the change, ask for changes, or even reject it.

Similarly, for SUSE Linux Enterprise openSUSE is an upstream development - and the openSUSE community with its release team can decide what to do with changes or requests coming from downstream (SUSE Linux Enterprise).

Also, SUSE as a company, has developers working in upstream projects like GNOME or the Linux kernel as well as in the upstream openSUSE and for sure on SUSE Linux Enterprise.

There are different ways to integrate: Some of that work will be done downstream first and then pushed upstream - and others will be done upstream with the intent to have it downstream later.

openSUSE 12.1 saw for example the integration of snapper - a snapshoting tool for BtrFS where development was driven by SUSE Linux Enterprise needs with the goal to have it in SUSE Linux Enterprise 11 Service Pack 2. It was stable, integrated without problems into openSUSE Factory and thus was added in time for openSUSE 12.1.

Note that SUSE Linux Enterprise as a distribution selects packages from openSUSE, adds some of its own, especially own branding packages, might have some different policies (e.g. how to partition or regarding security) and a different installation work flow. Before a release is done, the distribution need to run on all supported architectures - currently x86, x86-64, System z, Power, Itanium -, gets tested extensively which makes bugfixes necessary and gets certified by ISVs and IHVs and passes certifications like LSB or for IPv6 compliance. This testing leads to fixes that then go back to the upstreams - to the openSUSE distribution and the upstream open source projects.

A recent example: The testing for SLE 11 SP2 on Linux 3.0 resulted in a number of fixes. These went in the upstream kernel and thus became via upstream part of the Linux kernel that openSUSE uses. Many of these patches were also added directly to the openSUSE kernel.

So, SUSE Linux Enterprise is working in many ways similar with upstream as the openSUSE GNOME team with the upstream GNOME project: both take packages from upstream, add own branding, have refined polices from upstream, send patches etc.

What hasn't been done loudly in the past is raising voice - the SUSE Linux Enterprise developers and product managers raising their voice on what they need for their product and therefore would like to have from openSUSE and discuss how this can be done best. I expect to see more of this engagement of the openSUSE community by the SUSE Linux Enterprise stakeholder.

Recent changes in openSUSE Factory - Kernel and X.Org

2012-04-28T21:55:00.002+02:00

Just a quick note on two topics regarding the openSUSE distribution:

Kernel

Linus has released Linux 3.4 RC4 and you can get it as usual as RPM from the openSUSE Build Service in the Kernel:HEAD repository (download it here). it's expected that kernel 3.4 will be the openSUSE 12.2 kernel.
Jeff Mahoney has disabled a couple of options to make the resulting kernel image smaller and faster to build. He disabled DECNet and ARCnet and LocalTalk drivers on
i386-default. Also many drivers that only be used on embedded hardware are disabled. The last change was not done for ARM.
Full details are in his email message.

X.Org

Dominique and Vincent have updated X.Org to the current 1.12.1 release and all the packages are now available for testing (check this email for details). During the update they splitted each tar ball in his own package and also updated the metadata for it so that now "osc collab" (install the package osc-plugin-collab from openSUSE:Tools to use it) and the build service status view will show available upstream show version.

openSUSE's Freight Train

2012-04-27T10:24:00.002+02:00

A new year comes with new goals and plans. The year 2013 for SUSE starts on the 1st of April 2012 and one of the goals is to increase the amount of effort SUSE engineers put into openSUSE. So, we'll see soon in openSUSE more SUSE engineers actively contributing to Factory.

More active people means more conflicting needs and interests! To preserve our culture of "just do it" and "have fun", we've established a mail address where everyone (volunteer or not) can mail to if he/she feels there is a blockage in either the "doing" or the "fun". Such blockages can be documentation, procedures, technology and tools, or people.

A team of people who've been around for a while and know the technical as well as social side of openSUSE is there, dedicated to make sure you can do and have fun - currently: Henne Vogelsang, Jos Poortvliet and myself will take care of any complaints you send to them via freighttrain@opensuse.org. We'd also. For more details, read also the Freight Train wiki page. We also welcome more people to join our team - and hope that the freight train will not become a problem.

Why do we use the word "freight train" here? The openSUSE community consists of many different people - and some of them are able to work full-time on openSUSE, perhaps as part of a team of others. Those people might have a faster drive than others - and might "roll them over". We'd like to avoid these situations and whenever they happen to help fixing.

openSUSE Factory switched to GCC 4.7

2012-04-20T15:40:00.000+02:00

The GNU Compiler Collection (gcc) version 4.7 was released a month ago and comes with the usual optimzation improvements and changes to follow newer standards like the new C11 revision of the ISO C standard and the ISO C+11 standard. A full list of changes is available on the gcc wiki.

The openSUSE project has now changed the compiler settings so that by default gcc 4.7 is used. Over the last several weeks, a couple of engineers led by Dominique Leuenberger have used in the open build service a project where all packages of openSUSE Factory were compiled with gcc 4.7 and then those packages that did not build were fixed. A handful of packages needs still updating but AFAIK more than 100 packages were fixed to build with gcc 4.7 - and some were fixed to build again with current packages in Factory.
At least two packages triggered bugs in gcc, one of them is now fixed in upstream gcc and also our gcc package (thanks to the upstream gcc maintainer Richard Günther who is also the openSUSE gcc package maintainer), the other bug is still being investigated.

The switch to gcc 4.7 will now affect all the packages that built against factory but are not part of factory - once these get rebuild with the new gcc, the better standard conformance of the compiler, might lead to build failures.

For failing packages, the advise is to check first whether there's an upstream patch to fix the failure, then try fixing it yourself (often just a missing include), and then ask on the opensuse-factory mailing list or on the Freenode #opensuse-factory IRC channel for help.

If you like to test a package with several compilers, it's easy to do since openSUSE Factory currently includes a couple of GCC versions that can be installed in parallel.
The gcc package will install gcc 4.7 thus "gcc" is the command to use (or "gcc-4.7"), the gcc46 package contains a gcc 4.6 that can be used with "gcc-4.6" and there are also gcc41 and gcc43 packages. Another compiler that is available is llvm.

Btrfs Fixes for openSUSE

2012-04-02T20:21:00.000+02:00

openSUSE 12.1 is the first distribution that allows the creation of btrfs partitions in the installer. It also comes with a nice tool called snapper for managing snapshots of systems.

SUSE released a few weeks ago SUSE Linux Enterprise 11 Service Pack 2 which supports btrfs now as a file system and thus it's not a technology preview anymore. The SUSE Linux Enterprise team has done a lot of testing of btrfs and fixed many bugs - and backported btrfs fixes from the upstream kernels into the Linux 3.0 kernel that is used in the SUSE Linux Enterprise 11 SP2. openSUSE 12.1 comes with the newer Linux 3.1 kernel and the SUSE Labs kernel engineers have also added the same fixes that have been added to the enterprise kernel to this openSUSE kernel.

The 3.1.9 kernel which was released for openSUSE early February contained all fixes at that time. Additional fixes have been added since then and will be in the next update. These fixes include stability fixes and handling of errors.
For those that want to have the current kernel - before it gets released -, you can grab the kernel compiled daily from the kernel git repository by the openSUSE Build Service from one of our mirrors.

The openSUSE kernel team has more information about Linux kernels for openSUSE on its own web site http://kernel.opensuse.org.

I'd like to thank especially David Sterba, Jeff Mahoney and Mark Fashesh for debugging btrfs issues, fixing them and for backporting all the btrfs fixes to the openSUSE kernel!

/tmp as tmpfs for openSUSE?

2012-03-30T16:08:00.003+02:00

Frederic informed the openSUSE factory list that upstream systemd is going to use tmpfs for the /tmp filesystem by default soon. Lennart Poettering has posted a blog entry describing this and gives also some recommendations on how to use /tmp. Debian has recently made the switch and Fedora is considering it as well. A major benefit would be "By implementing this we, by default, generate less IO on disks. This increases SSD lifetime, saves a bit of power and makes things a bit faster."

A lot of openSUSE users argued that this is fine for systems with lots of memory - and some users already run their openSUSE installation with /tmp on tmpfs - but not good for tight memory since both tmpfs and applications will use the main memory.

A couple of ideas where discussed and it seems that openSUSE - at least for openSUSE 12.2 - will do the following:

For system daemons, we turn on PrivateTmp in systemd .services files where applicable so that each daemon has its own private tmp directory (for details see the Fedora proposal)
For users, we consider to give each user a separate $TMPDIR directory.

It should be possible to setup /tmp as tmpfs but the default for now should stay the same.

There's still some discussion going on and it might be that in the future openSUSE switches. But before that, we have to evaluate applications and how they use /tmp since many place large data there (e.g. gcc and binutils can easily place 4GB during linking there). Once it's clear where what kind of data belongs (see Lennart's blog entry for a recommendation) and packages are fixed, it's IMO time to reconsider this.
Would you want tmpfs on your system in the future?

New GNU C Library Version Coming to openSUSE

2012-01-04T17:50:00.000+01:00

The GNU C library (glibc) is the basic system library used by almost every program, it comes with the base library (libc), a math library (libm), the thread library (libpthread), support for locales and character conversions and a couple of tools. The latest release - 2.15 - was released Christmas 2011.
I have now updated glibc for openSUSE 12.2 in its development project (Base:System) to version 2.15 and it seems to work on x86, x86-64, PowerPC and even Arm.

The new version 2.15 of glibc brings mainly optimizations and bugfixes.

The main new features of the 2.15 version are:

a new program pldd to list loaded objects of a process.
nss_db support - without a dependency on berkely db.
integration of libm with gcc's -ffinite-math-only option.

Besides updating the upstream version, I made a couple of cleanups of the package itself:

The x86-64 package was falsely compiled with gcc's flag -mno-tls-direct-seg-refs - an option only needed for running under 32-bit Xen. This might speedup some threaded applications.
The i686 library is not compiled with -mno-tls-direct-seg-refs anymore but a special version is installed in /lib/i686/nosegneg that is compiled with this flag. So, installations using 32-bit Xen will use the nosegneg library and everybody else the normal version. This should speedup some threaded applications when not using Xen.
The AMD libm implementation has been removed in favour of glibc's libm implementation. It didn't integrate well with the changes that went into glibc 2.15.
I've reviewed all patches we have in the package and tried to upstream as many as possible - and removed a few that were obsolete. We still have far too many, help is welcome to upstream ;)

Call for testing
I welcome contributions to the package, testing of glibc 2.15 and fixing of any bugs in glibc or the packages using it. I'm especially interested on tests under Xen to check that my optimizations did not break anything.

So far, one package was failing due to changes - unscd - and the upstream author provided quickly a fix. Thanks to Cristian Rodriguez for his tests!

I'm pushing glibc 2.15 now to openSUSE:Factory for the usual review and integration.

Finding subtile malloc bugs

2012-01-03T17:17:00.000+01:00

Last year I've talked with a couple of engineers that had where hunting done some strange bug reports with memory corruption that upstream developers could not directly reproduce. The bugs included accessing data after a closedir call or buffer overflows. These kind of bugs normally lead to random behaviour at one point in time and are really difficult to debug. In the case of openSUSE Factory, the openSUSE developers could reproduceable trigger these since glibc comes with some helpers.
If "MALLOC_PERTURB_" is set in the environment, then every malloc call will clear memory after every free - thus destroying its content - and also initialize malloc'ed memory (with the exception of cmalloc).

Ulrich Drepper who implements this said:

"The reason for this exercise is, of course, to find code which uses memory returned by malloc without initializing it and code which uses code after it is freed. valgrind can do this but it's costly to run. The MALLOC_PERTURB_ exchanges the ability to detect problems in 100% of the cases with speed."

In openSUSE, we set MALLOC_PERTURB_ to 69. So, if you find your memory location suddenly is filled with "E"s (ASCII 69 is 'E') or with "\272" (the bitwise inverse), you have found a bug in handling of malloc.

For more information, read Ulrich's blog or Jakub's blog.

This is only enabled for openSUSE Factory but if you're developing software yourself, I advise to set it yourself and use it to find bugs early.