Our Cloud in Liberty

We have previously posted experiences with upgrades of OpenStack such as

• Kilo - https://openstack-in-production.blogspot.fr/2015/11/our-cloud-in-kilo.html
• Juno - http://openstack-in-production.blogspot.fr/2015/05/our-cloud-in-juno.html
• Icehouse - http://openstack-in-production.blogspot.fr/2014/11/our-cloud-in-icehouse.html
• Havana - http://openstack-in-production.blogspot.fr/2014/02/our-cloud-in-havana.html

The upgrade to Liberty for the CERN cloud was completed at the end of August. Working with the upstream OpenStack, Puppet and RDO communities, this went pretty smoothly without any issues so there is no significant advice to report. We followed the same approach as the past, gradually upgrading component by component.  With the LHC reaching it's highest data rates so far this year (over 10PB recorded to tape during June), the upgrades needed to be done without disturbing the running VMs.

Some hypervisors are still on Scientific Linux CERN 6 (Kilo) using Python 2.7 in a software collection but the backwards compatibility has allows the rest of the cloud to migrate while we complete the migration of 5000 VMs from old hardware and SLC6 to new hardware on CentOS 7 in the next few months.

After the migration, we did encounter a few problems:

• Live migration from Kilo to Liberty gave an error (https://bugs.launchpad.net/nova/+bug/1576048)
• Cannot delete machines with NUMA topology created before the upgrade (https://bugs.launchpad.net/nova/+bug/1596119)
• With the move to Nova API 2.1, a change in policy files was needed for our 'operator' role (details are at https://openstack-in-production.blogspot.ch/2015/02/delegation-of-roles.html). Some additional actions needed to be defined for the operator such as os_compute_api:servers:detail.

The first two cases are being backported to Liberty so others who have not upgraded may not see these.

We've already started the Mitaka upgrade with Barbican, Magnum and Heat ahead of the other components as we enable a CERN production container service. The other components will follow in the next six months including the experiences of the migration to Neutron which we'll share at the OpenStack summit in October.