tag:blogger.com,1999:blog-60328966651805592024-03-18T02:48:10.390-07:00OpenStack in Production - ArchivesNow moved to http://techblog.web.cern.chAnonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.comBlogger48125tag:blogger.com,1999:blog-6032896665180559.post-28986789966456705712019-01-19T01:31:00.000-08:002019-01-19T01:31:46.571-08:00OpenStack In Production - moving to a new home<br />
During 2011 and 2012, CERN IT took a new approach to how to manage the infrastructure for analysing the data from the LHC and other experiments. The Agile Infrastructure project was formed covering service provisioning, configuration management and monitoring by adopting commonly used open source solutions with active communities to replace the in house tool suite.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-mnMtT0bSYMM/XD3GgzfzqoI/AAAAAAAAVJ8/qVrL2HimvQU9R1G-Y9BsE5wPhIk45SWJgCLcBGAs/s1600/cernopenstackreleases.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="895" data-original-width="1600" height="223" src="https://3.bp.blogspot.com/-mnMtT0bSYMM/XD3GgzfzqoI/AAAAAAAAVJ8/qVrL2HimvQU9R1G-Y9BsE5wPhIk45SWJgCLcBGAs/s400/cernopenstackreleases.png" width="400" /></a></div>
<br />
<br />
In 2019, the CERN cloud managed infrastructure has grown by a factor of 10 compared to the resources in 2013. This has been achieved in collaboration with the many open source communities we have worked with over the past years, including<br />
<ul>
<li>OpenStack</li>
<li>RDO</li>
<li>CentOS</li>
<li>Puppet</li>
<li>Foreman</li>
<li>Elastic</li>
<li>Grafana</li>
<li>Collectd</li>
<li>Kubernetes</li>
<li>Ceph</li>
</ul>
<div>
These experiences have been shared with over 40 blogs and more than 100 different talks at open source events during this time from small user groups to large international conferences. </div>
<div>
<br /></div>
<div>
The OpenStack-In-Production blog has been covering the experiences, with a primary focus on the development of the CERN cloud service. However, the challenges of the open source world are now covering many more projects so it is time to move to a new blog to cover not only work on OpenStack but other communities and the approaches to integrated these projects into the CERN production infrastructure.</div>
<div>
<br /></div>
<div>
Thus, this blog will be moving to its new home at <a href="https://techblog.web.cern.ch/techblog/">https://techblog.web.cern.ch/techblog/</a>, incorporating our experiences with these other technologies. For those who would like to follow a specific subset of our activities, there are also taxonomy based content to select new OpenStack articles at <a href="https://techblog.web.cern.ch/techblog/tags/openstack/">https://techblog.web.cern.ch/techblog/tags/openstack/</a> and the legacy blog content at <a href="https://techblog.web.cern.ch/techblog/tags/openinfra/">https://techblog.web.cern.ch/techblog/tags/openinfra/</a>.</div>
<div>
<br /></div>
<div>
One of the most significant benefits we've found from sharing is receiving the comments from other community members. These often help to guide us in further investigations on solving difficult problems and to identify common needs to work on together upstream.<br />
<br />
Look forward to hearing from you all on the <a href="https://techblog.web.cern.ch/techblog/">techblog</a> web site.<br />
<br />
<h3>
References</h3>
</div>
<div>
<ul>
<li>CERN Cloud - 5 year perspective - <a href="https://www.slideshare.net/noggin143/20181219-ucc-open-stack-5-years-v3-128057833">https://www.slideshare.net/noggin143/20181219-ucc-open-stack-5-years-v3-128057833</a></li>
</ul>
<div>
<br /></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com93tag:blogger.com,1999:blog-6032896665180559.post-63598683884458424972018-05-09T10:55:00.000-07:002018-05-11T01:12:51.904-07:00Introducing GPUs to the CERN Cloud<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="margin-bottom: 1em; text-align: left;">
High-energy physics workloads can benefit from massive parallelism -- and as a matter of fact, the domain faces an increasing adoption of deep learning solutions. Take for example the newly-announced TrackML challenge <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[7]">[7]</a>, already running in Kaggle! This context motivates CERN to consider GPU provisioning in our OpenStack cloud, as <i>computation accelerators</i>, promising access to powerful GPU computing resources to developers and batch processing alike.</div>
<div>
<h3 style="text-align: left;">
What are the options?</h3>
</div>
<div>
Given the nature of our workloads, our focus is on discrete PCI-E Nvidia cards, like the GTX1080Ti and the Tesla P100. There are 2 ways to provision these GPUs to VMs: <i>PCI passthrough</i> and <i>virtualized GPU</i>. The first method is not specific to GPUs, but applies to any PCI device. The device is claimed by a generic driver, VFIO, on the hypervisor (which cannot use it anymore) and exclusive access to it is given to a single VM <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[1]">[1]</a>. Essentially, from the host’s perspective the VM becomes a userspace driver <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[2]">[2]</a>, while the VM sees the physical PCI device and can use normal drivers, expecting <i>no functionality limitation</i> and <i>no performance overhead</i>.</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-DL-SXn9ig1Q/WvMWbg9yx7I/AAAAAAAANbQ/jqgz4KLqz0gx16-Q5Q9ugP101dK2VX-qQCEwYBhgL/s1600/passthrough%252Bmdev.png" imageanchor="1" style="margin-left: auto; margin-right: auto;">
<img border="0" data-original-height="489" data-original-width="468" height="320" src="https://1.bp.blogspot.com/-DL-SXn9ig1Q/WvMWbg9yx7I/AAAAAAAANbQ/jqgz4KLqz0gx16-Q5Q9ugP101dK2VX-qQCEwYBhgL/s320/passthrough%252Bmdev.png" width="306" />
</a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Visualizing passthrough vs mdev vGPU <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[9]">[9]</a>
</td></tr>
</tbody></table>
<div style="margin-bottom: 1em; text-align: left;">
In fact, perhaps some “limitation in functionality” is warranted, so that the untrusted VM can’t do low-level hardware configuration changes on the passed-through device, like changing power settings or even its firmware! In fact, security-wise PCI passthrough leaves a lot to be desired. Apart from allowing the VM to change the device’s configuration, it might leave a possibility for side-channel attacks on the hypervisor (although we have not observed this, and a hardware “IOMMU” protects against DMA attacks from the passed-through device). Perhaps more importantly, the device’s state won’t be automatically reset after deallocating from a VM. In the case of a GPU, data from a previous use may persist on the device’s global memory when it is allocated to a new VM. The first concern may be mitigated by improving VFIO, while the latter, the issue of device reset or “cleanup”, provides a use case for a more general accelerator management framework in OpenStack -- the nascent Cyborg project may fit the bill.</div>
<div style="margin-bottom: 1em; text-align: left;">
Virtualized GPUs are a vendor-specific option, promising better manageability and alleviating the previous issues. Instead of having to pass through entire physical devices, we can split physical devices into virtual pieces on demand (well, almost on demand; there needs to be no vGPU allocated in order to change the split) and hand out a piece of GPU to any VM. This solution is indeed more elegant. In Intel and Nvidia’s case, virtualization is implemented as a software layer in the hypervisor, which provides “mediated devices” (mdev <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[3]">[3]</a>), virtual slices of GPU that appear like virtual PCI devices to the host and can be given to the VMs individually. This requires a special vendor-specific driver on the hypervisor (Nvidia GRID, Intel GVT-g), unfortunately not yet supporting KVM. AMD is following a different path, implementing SR-IOV at a hardware level.</div>
<h3 style="text-align: left;">
CERN’s implementation</h3>
<div style="margin-bottom: 1em; text-align: left;">
PCI passthrough has been supported in Nova for several releases, so it was the first solution we tried. There is a guide in the OpenStack docs <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[4]">[4]</a>, as well as previous summit talks on the subject <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[1]">[1]</a>. Once everything is configured, the users will see special VM flavors (“g1.large”), whose extra_specs field includes passthrough of a particular kind of gpu. For example, to deploy a GTX 1080Ti, we use the following configuration:</div>
<div>
<table align="left">
<tbody>
<tr><td><table align="left" style="border-top: 1px solid black; text-align: left; width: 100%;">
<tbody>
<tr><th>nova-compute</th></tr>
<tr><td><code>pci_passthrough_whitelist={"vendor_id":"10de"}</code></td></tr>
</tbody></table>
</td></tr>
<tr><td><table align="left" style="border-top: 1px solid black; text-align: left; width: 100%;">
<tbody>
<tr><th>nova-scheduler</th></tr>
<tr><td>add <code>PciPassthroughFilter</code> to enabled/default filters</td></tr>
</tbody></table>
</td></tr>
<tr><td><table align="left" style="border-top: 1px solid black; text-align: left; width: 100%;">
<tbody>
<tr><th>nova-api</th></tr>
<tr>
</tr>
<tr><td><code>pci_alias={"vendor_id":"10de",”product_id”:”1b06”,”device_type”:”type-PCI”,”name”:”nvP1080ti_VGA”}</code></td></tr>
<tr><td><code>pci_alias={"vendor_id":"10de",”product_id”:”10ef”,”device_type”:”type-PCI”,”name”:”nvP1080ti_SND”}</code></td></tr>
</tbody></table>
</td></tr>
<tr><td><table align="left" style="border-bottom: 1px solid black; border-top: 1px solid black; text-align: left; width: 100%;">
<tbody>
<tr><th>flavor extra_specs</th></tr>
<tr>
</tr>
<tr><td><code>--property "pci_passthrough:alias"="nvP1080ti_VGA:1,nvP1080ti_SND:1"</code></td></tr>
</tbody></table>
</td></tr>
</tbody></table>
</div>
<div style="margin-bottom: 1em; text-align: left;">
A detail here is that most GPUs appear as 2 pci devices, the VGA and the sound device, both of which must be passed through at the same time (they are in the same IOMMU group; basically an IOMMU group <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[6]">[6]</a> is the smallest passable unit).
</div>
<div style="margin-bottom: 1em; text-align: left;">
Our cloud was in Ocata at the time, using CellsV1, and there were a few hiccups, such as the Puppet modules not parsing an option syntax correctly (MulitStrOpt) and CellsV1 dropping the pci requests. For Puppet, we were simply missing some upstream commits <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[15]">[15]</a>. From Pike on and in CellsV2, these issues shouldn’t exist. As soon as we had worked around them and puppetized our hypervisor configuration, we started offering cloud GPUs with PCI passthrough and evaluating the solution. We created a few GPU flavors, following the AWS example of keeping the amount of vCPUs the same as the corresponding normal flavors.</div>
<div style="margin-bottom: 1em; text-align: left;">
From the user’s perspective, there proved to be no functionality issues. CUDA applications, like TensorFlow, run normally; the users are very happy that they finally have exclusive access to their GPUs (there is good tenant isolation). And there is no performance penalty in the VM, as measured by the SHOC benchmark <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[5]">[5]</a> -- admittedly quite old, we preferred this benchmark because it also measures low-level details, apart from just application performance.</div>
<div style="margin-bottom: 1em; text-align: left;">
From the cloud provider’s perspective, there’s a few issues. Apart from the potential security problems identified before, since the hypervisor has no control over the passed-through device, we can’t monitor the GPU. We can’t measure its actual utilization, or get warnings in case of critical events, like overheating.</div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-X5F9NChzoao/WvMt9eYx5II/AAAAAAAANbs/NXDajXWc8TkCXm1SzXa6fOSB3PIriV-xwCEwYBhgL/s1600/GPU%2BMemory%2BBandwidth.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="535" data-original-width="764" height="224" src="https://3.bp.blogspot.com/-X5F9NChzoao/WvMt9eYx5II/AAAAAAAANbs/NXDajXWc8TkCXm1SzXa6fOSB3PIriV-xwCEwYBhgL/s320/GPU%2BMemory%2BBandwidth.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-6_vJ7yTsYrA/WvMt9UE5afI/AAAAAAAANbw/C1YkAzUlbJAYaz6rqu1T07fVWCqHrVoVgCEwYBhgL/s1600/MaxFlops.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="535" data-original-width="764" height="224" src="https://3.bp.blogspot.com/-6_vJ7yTsYrA/WvMt9UE5afI/AAAAAAAANbw/C1YkAzUlbJAYaz6rqu1T07fVWCqHrVoVgCEwYBhgL/s320/MaxFlops.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-xoE8MTy6BcE/WvM06BPJ7MI/AAAAAAAANcM/oGzelpKoOz8lyjj-OlQMzUZHVVSx0G-9wCLcBGAs/s1600/GEMM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="535" data-original-width="764" height="224" src="https://2.bp.blogspot.com/-xoE8MTy6BcE/WvM06BPJ7MI/AAAAAAAANcM/oGzelpKoOz8lyjj-OlQMzUZHVVSx0G-9wCLcBGAs/s320/GEMM.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-jdEv53OXUMg/WvMt96YT_GI/AAAAAAAANbw/_x4YaZuGNwU4fUaqns64ASdGo_jJ2QE4wCEwYBhgL/s1600/S3D.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="535" data-original-width="764" height="224" src="https://4.bp.blogspot.com/-jdEv53OXUMg/WvMt96YT_GI/AAAAAAAANbw/_x4YaZuGNwU4fUaqns64ASdGo_jJ2QE4wCEwYBhgL/s320/S3D.png" width="320" /></a></div>
</td></tr>
<tr><td class="tr-caption" style="text-align: center;">Normalized performance of VMs vs. hypervisor on some SHOC benchmarks. First 2: low-level gpu features, Last 2: gpu algorithms <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[8]">[8]</a>. There are different test cases of VMs, to check if other parameters play a role. The “Small VM” has 2 vCPUs, “Large VM” has 4, “Pinned VM” has 2 pinned vCPUs (thread siblings), “2 pin diff N” and “2 pin same N” measure performance in 2 pinned VMs running simultaneously, in different vs the same NUMA nodes
</td></tr>
</tbody></table>
</div>
<h3 style="text-align: left;">
Virtualized GPU experiments</h3>
<div style="margin-bottom: 1em; text-align: left;">
The allure of vGPUs amounts largely to finer-grained distribution of resources, less security concerns (debatable) and monitoring. Nova support for provisioning vGPUs is offered in Queens as an experimental feature. However, our cloud is running on KVM hypervisors (on CERN CentOS 7.4 <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[14]">[14]</a>), which Nvidia does not support as of May 2018 <span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">(Nvidia GRID v6.0)</span>. When it does, the hypervisor will be able to split the GPU into vGPUs according to one of many possible profiles, such as in 4 or in 16 pieces. Libvirt then assigns these mdevs to VMs in a similar way to hostdevs (passthrough). Details are in the OpenStack docs at <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[16]">[16]</a>.</div>
<div style="margin-bottom: 1em; text-align: left;">
Despite this promise, it remains to be seen if virtual GPUs will turn out to be an attractive offering for us. This depends on vendors’ licensing costs (such as per VM pricing), which, for the compute-compatible offering, can be significant. Added to that is the fact that only a subset of standard CUDA is supported (<i>not</i> supported are the unified memory and “CUDA tools” <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[11]">[11]</a>, probably referring to tools like the Nvidia profiler). vGPUs are also oversubscribing the GPU’s compute resources, which can be seen in either a positive or negative light. On the one hand, this guarantees higher resource utilization, especially for bursting workloads, like developers. On the other hand, we may expect a lower quality of service <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[12]">[12]</a>.</div>
<h3 style="text-align: left;">
And the road goes on...</h3>
<div style="margin-bottom: 1em; text-align: left;">
Our initial cloud GPU offering is very limited, and we intend to gradually increase it. Before that, it will be important to address (or at least be conscious about) the security repercussions of PCI passthrough. But even more significant is to address GPU accounting in a straightforward manner, by enforcing quotas on GPU resources. So far we haven’t tested the case of GPU P2P, with multi-GPU VMs, which is supposed to be problematic <a href="https://openstack-in-production.blogspot.fr/2018/05/introducing-gpus-to-cern-cloud.html#[13]">[13]</a>.</div>
<div style="margin-bottom: 1em; text-align: left;">
Another direction we’ll be researching is offering GPU-enabled container clusters, backed by pci-passthrough VMs. It may be that, with this approach, we can emulate a behavior similar to vGPUs and circumvent some of the bigger problems with pci passthrough.</div>
<div>
<h3 style="text-align: left;">
References</h3>
</div>
<div>
<div id="[1]">
[1]: OVH presentation: <a href="https://www.youtube.com/watch?v=1tdvz3ejokM">https://www.youtube.com/watch?v=1tdvz3ejokM</a></div>
<div id="[2]">
[2]: VFIO description: <a href="https://www.kernel.org/doc/Documentation/vfio.txt">https://www.kernel.org/doc/Documentation/vfio.txt</a></div>
<div id="[3]">
[3]: VFIO mediated devices (mdev): <a href="https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt">https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt</a></div>
<div id="[4]">
[4]: Nova docs on PCI passthrough: <a href="https://docs.openstack.org/nova/latest/admin/pci-passthrough.html">https://docs.openstack.org/nova/latest/admin/pci-passthrough.html</a></div>
<div id="[5]">
[5]: SHOC benchmark suite: <a href="https://github.com/vetter/shoc">https://github.com/vetter/shoc</a></div>
<div id="[6]">
[6]: IOMMU groups: <a href="https://vfio.blogspot.ch/2014/08/iommu-groups-inside-and-out.html">https://vfio.blogspot.ch/2014/08/iommu-groups-inside-and-out.html</a></div>
<div id="[7]">
[7]: TrackML challenge announcement: <a href="https://home.cern/about/updates/2018/05/are-you-trackml-challenge">https://home.cern/about/updates/2018/05/are-you-trackml-challenge</a></div>
<div id="[8]">
[8]: S3D: <a href="https://github.com/vetter/shoc/wiki/S3d">https://github.com/vetter/shoc/wiki/S3d</a>, <a href="https://www.olcf.ornl.gov/wp-content/themes/olcf/titan/Titan_BuiltForScience.pdf">https://www.olcf.ornl.gov/wp-content/themes/olcf/titan/Titan_BuiltForScience.pdf</a></div>
<div id="[9]">
[9]: Images taken from: <a href="http://www.linux-kvm.org/images/5/59/02x03-Neo_Jia_and_Kirti_Wankhede-vGPU_on_KVM-A_VFIO_based_Framework.pdf">http://www.linux-kvm.org/images/5/59/02x03-Neo_Jia_and_Kirti_Wankhede-vGPU_on_KVM-A_VFIO_based_Framework.pdf</a></div>
<div id="[10]">
[10]: Nvidia vGPU architecture: <a href="https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#architecture-grid-vgpu">https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#architecture-grid-vgpu</a></div>
<div id="[11]">
[11]: CUDA Unified memory and tooling not supported on Nvidia vGPU: <a href="https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#features-grid-vgpu">https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#features-grid-vgpu</a></div>
<div id="[12]">
[12]: AMD on vGPU QoS: <a href="https://pro.radeon.com/en/quality-of-service-amd-mxgpu/">https://pro.radeon.com/en/quality-of-service-amd-mxgpu/</a></div>
<div id="[13]">
[13]: GPU P2P doesn’t work: <a href="http://lists.openstack.org/pipermail/openstack-operators/2018-March/014988.html">http://lists.openstack.org/pipermail/openstack-operators/2018-March/014988.html</a></div>
<div id="[14]">
[14]: CERN CentOS 7: <a href="https://linux.web.cern.ch/linux/centos7/">https://linux.web.cern.ch/linux/centos7/</a></div>
<div id="[15]">
[15]: Missing Puppet commits: <a href="https://github.com/openstack/puppet-nova/commit/e7fe8c16ae873834ccf145b2bcbc62081a957241">https://github.com/openstack/puppet-nova/commit/e7fe8c16ae873834ccf145b2bcbc62081a957241</a>, <a href="https://github.com/openstack/puppet-nova/commit/c1a4ab211dd2322572349719379cd13c6f2abb9a">https://github.com/openstack/puppet-nova/commit/c1a4ab211dd2322572349719379cd13c6f2abb9a</a></div>
<div id="[16]">
[16]: Adding vGPUs to guests: <a href="https://docs.openstack.org/nova/latest/admin/virtual-gpu.html">https://docs.openstack.org/nova/latest/admin/virtual-gpu.html</a></div>
<div>
<br /></div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/13444253716218476471noreply@blogger.com23tag:blogger.com,1999:blog-6032896665180559.post-13871819823507244982018-03-12T04:29:00.001-07:002018-03-12T04:29:28.031-07:00Hardware burn-in in the CERN datacenter<br />
During the Ironic sessions at the recent OpenStack Dublin PTG in Spring 2018, there were some discussions on adding a further burn in step to the OpenStack Bare Metal project (Ironic) <a href="https://docs.openstack.org/ironic/pike/contributor/states.html">state machine</a>. The notes summarising the sessions were reported to the <a href="https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg116584.html">openstack-dev</a> list. This blog covers the CERN burn in process for the systems delivered to the data centers as one example of how OpenStack Ironic users could benefit from a set of open source tools to burn in newly delivered servers as a stage within the Ironic workflow.<br />
<br />
CERN hardware procurement follows a<a href="http://procurement.web.cern.ch/en/procurement-strategy-and-policy"> formal process compliant</a> with public procurements. Following a market survey to identify potential companies in CERN's member states, a tender specification is sent to the companies asking for offers based on technical requirements.<br />
<h2>
Server burn in goals</h2>
<div>
Following the public procurement processes at CERN, large hardware deliveries occur once or twice a year and smaller deliveries multiple times per year. The overall resource management at CERN was covered in a previous <a href="http://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html">blog</a>. Part of the steps before production involves burn in of new servers. The goals are</div>
<div>
<ul>
<li>Ensure that the hardware delivered complies with CERN Technical Specifications</li>
<li>Find systematic issues with all machines in a delivery such as bad firmware</li>
<li>Identify failed components in single machines</li>
<li>Provoke early failure in failing components due to high load during stress testing</li>
</ul>
<div>
Depending on the hardware configuration, the burn-in tests take on average around two weeks but do vary significantly (e.g. for systems with large memory amounts, the memory tests alone can take up to two weeks). This has been found to be a reasonable balance between achieving the goals above compared to delaying the production use of the machines with further testing which may not find more errors.</div>
</div>
<div>
<br /></div>
<div>
Successful execution of the CERN burn in processes is required in the tender documents prior to completion of the invoicing.</div>
<h2>
Workflow</h2>
<div>
The CERN hardware follows a lifecycle from procurement to retirement as outlined below. The parts marked in red are the ones currently being implemented as part of the CERN Bare Metal deployment.</div>
<div>
<br /></div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-GgOlP-L4p0c/WqZhrDAQa4I/AAAAAAAAUuY/2wXS9uSK42k36lGgpFRVNddWAvn96jwYwCLcBGAs/s1600/FPP_Workflow_for_IRONIC_v3crop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="950" data-original-width="1600" height="378" src="https://1.bp.blogspot.com/-GgOlP-L4p0c/WqZhrDAQa4I/AAAAAAAAUuY/2wXS9uSK42k36lGgpFRVNddWAvn96jwYwCLcBGAs/s640/FPP_Workflow_for_IRONIC_v3crop.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
As part of the evaluation, test systems are requested from the vendor and these are used to validate compliance with the specifications. The results are also retained to ensure that the bulk equipment deliveries correspond to the initial test system configurations and performance.</div>
<h2>
Preliminary Checks</h2>
<div>
CERN requires that the Purchase Order ID and an unique System Serial Number are set in the NVRAM of the Baseboard Management Controller (BMC), in the Field Replaceable Unit (FRU) fields Product Asset Tag (PAT) and Product Serial (PS) respectively:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># ipmitool fru print 0 | tail -2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Product Serial : 245410-1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Product Asset Tag : CD5792984</span><br />
<br /></div>
<div>
The Product Asset Tag is set to the CERN delivery number and the Product Serial is set to the unique serial number for the system unit.</div>
<div>
<br /></div>
<div>
Likewise, certain BIOS fields have to be set correctly such as booting from network before disk to ensure the systems can be easily commissioned.</div>
<div>
<br /></div>
<div>
Once these basic checks have been done, the burn in process can start. A configuration file, containing the burn-in tests to be run, is created according on the information stored in the PAT and PS FRU fields. Based on the content of the configuration file, the enabled tests will automatically start.</div>
<h2>
Burn in</h2>
<div>
The burn in process itself is highlighted in red in the workflow above, consisting of the following steps</div>
<ul>
<li>Memory</li>
<li>CPU</li>
<li>Storage</li>
<li>Benchmarking</li>
<li>Network</li>
</ul>
<h3>
Memory</h3>
<div>
The memtest stress tester is used for validation of the RAM in the system. Details of the tool are available at <a href="http://www.memtest.org/">http://www.memtest.org/</a>. </div>
<h3>
CPU</h3>
<div>
Testing the CPU is performed using a set of burn tools, burnK7 or burnP6, and burn MMX. These tools not only test the CPU itself but are also useful to find cooling issues such as broken fans since the power load is significant with the processors running these tests.</div>
<h3>
Disk</h3>
<div>
Disk burn ins are intended to create the conditions for early drive failure. The <a href="https://www.datarecoup.com/blog/data-recovery/the-bathtub-curve">bathtub curve</a> aims to cause the early failure drives to fail prior to production.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-1PoikJv0mVQ/WqVRcyYkGnI/AAAAAAAAUt0/Q_1VaIlsZFUyrK_uLAndrJ45xzZMOpJ9wCLcBGAs/s1600/b2ap3_thumbnail_bathtub-curve.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="233" data-original-width="400" height="370" src="https://2.bp.blogspot.com/-1PoikJv0mVQ/WqVRcyYkGnI/AAAAAAAAUt0/Q_1VaIlsZFUyrK_uLAndrJ45xzZMOpJ9wCLcBGAs/s640/b2ap3_thumbnail_bathtub-curve.jpg" width="640" /></a></div>
<div>
<br /></div>
<div>
<div>
<br /></div>
<div>
With this aim, we use the <a href="https://en.wikipedia.org/wiki/Badblocks">badblocks</a> code to repeatedly read/write the disks. SMART counters are then checked to see if there are significant numbers of relocated bad blocks and the CERN tenders require disk replacement if the error rate is high.</div>
</div>
<div>
<br /></div>
<div>
We still use this process although the primary disk storage for the operating system has now changed to SSD. There may be a case for minimising the writing on an SSD to maximise the life cycle of the units.</div>
<h3>
Benchmarking</h3>
<div>
Many of the CERN hardware procurements are based on price for total compute capacity needed. With the nature of most of the physics processing, the total throughput of the compute farm is more important than the individual processor performance. Thus, it may be that the most total performance can be achieved by choosing processors which are slightly slower but less expensive.</div>
<div>
<br /></div>
<div>
CERN currently measures the CPU performance using a set of benchmarks based on a subset of the SPEC 2006 suite. The subset, called HEPSpec06, is run in parallel on each of the cores in the server to determine the total throughput from the system. Details are available at the <a href="http://w3.hepix.org/benchmarking.html">HEPiX Benchmarking Working Group</a> web site.</div>
<div>
<br /></div>
<div>
Since the offers include the expected benchmark performance, the results of the benchmarking process are used to validate the technical questionnaire submitted by the vendors. All machines in the same delivery would be expected to produce similar results so variations between different machines in the same batch are investigated.</div>
<div>
<br /></div>
<div>
CPU benchmarking can also be used to find problems where there is significant difference across a batch, such as incorrect BIOS settings on a particular system.</div>
<div>
<br /></div>
<div>
Disk performance is checked using a reference <a href="http://fio.readthedocs.io/en/latest/fio_doc.html">fio</a> access suite. A minimum performance level in I/O is also required in the tender documents.</div>
<div>
<br /></div>
<h3>
Networking</h3>
<div>
Networking interfaces are difficult to burn in compared to disks or CPU. To do a reasonable validation, at lest two machines are needed. With batches of 100s of servers, a simple test against a single end point will produce unpredictable results.</div>
<div>
<br /></div>
<div>
Using a network broadcast, the test finds other machines running the stress test, they pair up and run a number of tests.<br />
<br />
<ul>
<li><a href="https://iperf.fr/">iperf3</a> is used for bandwidth, reversed bandwidth, udp and reversed udp</li>
<li>iperf for full duplex testing (currently missing from iperf3)</li>
<li>ping is used for congestion testing</li>
</ul>
</div>
<h2>
Looking forward</h2>
<div>
CERN is currently deploying Ironic into production for bare metal management of machines. Integrating the burn in and retirement stages into the bare metal management states would bring easy visibility of the current state as the deliveries are processed.</div>
<div>
<br /></div>
<div>
The retirement stage is also of interest to ensure that there is no CERN configuration in the servers (such as Ironic BMC credentials or IP addresses). CERN has often donated retired servers to other high energy physics sites such as <a href="https://home.cern/cern-people/updates/2017/10/servers-sesame">SESAME in Jordan</a> and <a href="https://www.sharing-knowledge.org/the-cern-computer-centre-has-donated-161-retired-servers-to-universities-in-morocco/">Morocco</a> which requires a full server factory reset before dismounting. This retirement step would be a more extreme cleaning followed by complete removal from the cloud.</div>
<div>
<br /></div>
<div>
Discussing with other scientific laboratories such as <a href="https://www.skatelescope.org/">SKA</a> through the <a href="https://wiki.openstack.org/wiki/Scientific_SIG">OpenStack Scientific special interest group</a> has shown interest in extending Ironic to automate the server on-boarding and retirement processes as described in the session at the <a href="https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska">OpenStack Sydney summit</a>. We'll be following up on these discussions at Vancouver.</div>
<h2>
Acknowledgements</h2>
<div>
<ul>
<li>CERN IT department - <a href="http://cern.ch/it">http://cern.ch/it</a></li>
<li>CERN Ironic and Rework Contributors </li>
<ul>
<li>Alexandru Grigore</li>
<li>Daniel Abad</li>
<li>Mateusz Kowalski</li>
</ul>
</ul>
</div>
<h2>
References</h2>
<div>
<ul>
<li>Computing in High Energy Physics procurement paper (CHEP 2013) - <a href="http://iopscience.iop.org/article/10.1088/1742-6596/513/6/062003/pdf">http://iopscience.iop.org/article/10.1088/1742-6596/513/6/062003/pdf</a></li>
<li>Automatic server registration and burn-in framework (HEPiX Fall 2013) - <a href="https://indico.cern.ch/event/247864/contributions/1570349/attachments/426708/592287/Automatic_Registration_Burn_in.ppt">https://indico.cern.ch/event/247864/contributions/1570349/attachments/426708/592287/Automatic_Registration_Burn_in.ppt</a></li>
<li>Backblaze failure report - <a href="https://www.backblaze.com/blog/hard-drive-stats-for-2017/">https://www.backblaze.com/blog/hard-drive-stats-for-2017/</a></li>
<li>Resource Management at CERN - <a href="http://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html">http://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html</a></li>
</ul>
</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com18tag:blogger.com,1999:blog-6032896665180559.post-75751174846739281422018-03-02T03:46:00.002-08:002018-03-09T07:11:34.312-08:00Expiry of VMs in the CERN cloudThe CERN cloud resources are used for a variety of purposes from running compute intensive workloads to long running services. The cloud also provides personal projects for each user who is registered to the service. This allows a small quota (5 VMs, 10 cores) where the user can have resources dedicated for their use such as boxes for testing. A typical case would be for the CERN IT Tools training where personal projects are used as sandboxes for trying out tools such as Puppet.<br />
<br />
Personal projects have a number of differences compares to other projects in the cloud<br />
<ul>
<li>No non-standard flavors</li>
<li>No additional quota can be requested</li>
<li>Should not be used for production services</li>
<li>VMs are deleted automatically when the person stops being a CERN user</li>
</ul>
<div>
With the number of cloud users increasing to over 3,000, there is a corresponding growth in the number of cores used by personal projects, growing by 1,200 cores in the past year. For cases like training users, there is often the case that the VMs are created and the user then does not remember to delete the resources so they consume cores which could be used for compute capacity to analyse the data from the LHC.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-UNngAR17x2U/WpgR5oaR9xI/AAAAAAAAUpU/CxiBN9fqdq4FoNrse2viyA0lRJ7T_nS2QCLcBGAs/s1600/personalcores.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="648" data-original-width="1600" height="258" src="https://2.bp.blogspot.com/-UNngAR17x2U/WpgR5oaR9xI/AAAAAAAAUpU/CxiBN9fqdq4FoNrse2viyA0lRJ7T_nS2QCLcBGAs/s640/personalcores.png" width="640" /></a></div>
<div>
<br /></div>
<br />
One possible approach would be to reduce the quota further. However, tests such as setting up a Kubernetes cluster with OpenStack Magnum often need several VMs to perform the different roles so this would limit the usefulness of personal projects. The usage of the full quota is also rare.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-AqfFcpBGqSc/WpkCxszwnBI/AAAAAAAAUpk/Foc7StUo-9cSaX1DyXMPuEXFsTnXSBnpQCLcBGAs/s1600/vmsused.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="575" data-original-width="968" height="380" src="https://3.bp.blogspot.com/-AqfFcpBGqSc/WpkCxszwnBI/AAAAAAAAUpk/Foc7StUo-9cSaX1DyXMPuEXFsTnXSBnpQCLcBGAs/s640/vmsused.png" width="640" /></a></div>
<br />
<br />
<h2>
VM Expiration</h2>
<div>
Based on a previous service which offered resources on demand (called CVI based on Microsoft SCVMM), the approach was taken to expire personal virtual machines.</div>
<div>
<ul>
<li>Users can create virtual machines up to the limit of their quota</li>
<li>Personal VMs are marked with an expiry date</li>
<li>Prior to their expiry, the user is sent several mails to inform them their VM will expire soon and how to extend it if it is still useful.</li>
<li>On expiry, the virtual machine is locked and shutdown. This helps to catch cases where people have forgotten to prolong their VMs.</li>
<li>One week later, the virtual machine is deleted, freeing up the resources.</li>
</ul>
</div>
<h2>
Implementation</h2>
<div>
We use <a href="https://wiki.openstack.org/wiki/Mistral">Mistral </a>to automate several OpenStack tasks in the cloud (such as <a href="http://openstack-in-production.blogspot.fr/2017/08/scheduled-snapshots.html">regular snapshots</a> and project creation/deletion). This has the benefit of a clean audit log to show what steps worked/failed along with clear input/output states supporting retries and an authenticated cloud cron for scheduling.<br />
<br /></div>
<div>
Our OpenStack projects have some properties set when they are created. This is used to indicate additional information like the accounting codes to be charged for the usage. There are properties for indicating if the type of project such as personal and if the expiration workflow should apply. Mistral <a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/blob/master/workbooks/instance_expiration_global.yaml">YAQL code</a> can then select resources where expiration applies.<br />
<br />
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-GB</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="375">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hashtag"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Unresolved Mention"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073697537 9 0 511 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-GB;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-GB;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-GB;}
</style>
<![endif]-->
<!--StartFragment--><span lang="EN-GB" style="font-family: "calibri" , sans-serif; font-size: 12.0pt;">t</span><span lang="EN-GB" style="font-size: 12pt;"><span style="font-family: "courier new" , "courier" , monospace;">ask(retrieve_all_projects).result.select(dict(id
=> $.id, name => $.name, enabled => $.enabled, type =>
$.get('type','none'),expire =>
$.get('expire','off'))).where($.type='personal').where($.enabled).where($.expire='on')</span></span><!--EndFragment-->
</div>
<div>
<br /></div>
<div>
The <span style="font-family: "courier new" , "courier" , monospace;">expire_at</span> parameter is stored as a VM property. This makes it visible for automation such as CLIs through the openstack client show server CLI.</div>
<div>
<br /></div>
<div>
There are several parts to the process</div>
<div>
<ul>
<li>A cron trigger'd workflow which</li>
<ul>
<li>Machines in error state or currently building are ignored</li>
<li>A newly created machine which does not have an expiry date set has the expiration date set according to the grace period</li>
<li>Sees if any machines are entering close to their expiry time and sends a mail to the owner</li>
<li>Checks for invalid settings of the <span style="font-family: "courier new" , "courier" , monospace;">expire_at</span> property (such as people setting it a long way in the future or deleting the property) and restores a reasonable value if this is detected</li>
<li>If a machine has reached it's expiry date, it's locked and shutdown</li>
<li>If a machine has past it's date by the grace period, it's deleted</li>
</ul>
<ul>
</ul>
<li>A workflow, launched by Horizon or from the CLI</li>
<ul>
<li>Retrieves the <span style="font-family: "courier new" , "courier" , monospace;">expire_at</span> value and extends it by the prolongation period</li>
</ul>
</ul>
<div>
The user notification is done using a set of mail templates and a dedicated workflow (<a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/blob/master/workflows/send_mail_template.yaml">https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/blob/master/workflows/send_mail_template.yaml</a>). This allows templates such as <a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/blob/master/templates/instance_expiration_reminder_mail.yaml">instance reminders</a> to have details about the resources included, such as the example from the mail template.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">The Virtual Machine {instance} from the project {project_name} </span><span style="font-family: "courier new" , "courier" , monospace;">in the Cloud Infrastructure Service will expire on {expire_date}.</span></div>
</div>
<div>
<br /></div>
<div>
A couple of changes to Mistral will be submitted upstream</div>
</div>
<div>
<ul>
<li>Support for HTML mail bodies which allows us to have a nicer looking e-mail for notification with links included</li>
<li>Support for BCC/CC on the mail so that the OpenStack cloud administrator e-mail can also be kept on copy when there are notifications</li>
</ul>
</div>
<div>
A few minor changes to Horizon were also done (currently local patches)</div>
<div>
<ul>
<li>Display <span style="font-family: "courier new" , "courier" , monospace;">expire_at</span> value on the instance details page</li>
<li>Add a 'prolong' action so that instances can be prolonged via the web by using the properties editor to set the date of the expiry (defaulting to the current date with the expiry time). This launches the workflow for prolonging the instance.</li>
</ul>
</div>
<h2>
Author</h2>
<div>
Jose Castro Leon from the CERN cloud team</div>
<h2>
References</h2>
<div>
<ul>
<li>CERN IT department at <a href="http://information-technology.web.cern.ch/">http://information-technology.web.cern.ch/</a></li>
<li>Workflow code is in the repository at <a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows">https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows</a>, specifically at <a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/tree/master/workbooks">https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/tree/master/workbooks</a> for the YAQL.</li>
<li>CERN end user documentation is at <a href="http://clouddocs.web.cern.ch/clouddocs/projects/vm_expiration.html">http://clouddocs.web.cern.ch/clouddocs/projects/vm_expiration.html</a></li>
<li>OpenStack Mistral Workflow as a Service documentation is at <a href="https://docs.openstack.org/mistral/latest/overview.html">https://docs.openstack.org/mistral/latest/overview.html</a></li>
</ul>
<div>
<br /></div>
</div>
<br />
<br />Anonymoushttp://www.blogger.com/profile/09376084384626279065noreply@blogger.com69tag:blogger.com,1999:blog-6032896665180559.post-28982725924807647042018-02-21T04:37:00.000-08:002018-04-17T01:37:25.034-07:00Maximizing resource utilization with Preemptible Instances<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<b>Motivation</b></h2>
<br />
The CERN cloud consists of around 8,500 hypervisors providing over 36,000<br />
virtual machines. These provide the compute resources for both the laboratory's<br />
physics program but also for the organisation's administrative operations such<br />
as paying bills and reserving rooms at the hostel.<br />
<br />
The resources themselves are generally ordered once to twice a year with servers being kept for around 5 years. Within the CERN budget, the resource planning teams looks at:<br />
<ul style="text-align: left;">
<li>The needs of the physics program for the coming years under the review of the <a href="http://wlcg.web.cern.ch/collaboration/management/computing-resources-scrutiny-group">Computing Scrutiny Review Board</a> and the <a href="http://committees.web.cern.ch/committees/lhcc/">LHC Experiments Committee</a>.</li>
</ul>
<ul style="text-align: left;">
<li>The resources required to run the computing services requirements for the CERN laboratory. These are projected using capacity planning trend data and upcoming projects such as video conferencing.</li>
</ul>
With the installation and commissioning of thousands of servers concurrently<br />
(along with their associated decommissioning 5 years later), there are scenarios<br />
to exploit underutilised servers. Programs such as <a href="http://lhcathome.web.cern.ch/">LHC@Home</a> are used but we have also been interested to expand the cloud to provide virtual machine instances which can be rapidly terminated in the event of<br />
<ul style="text-align: left;">
<li>Resources being required for IT services as they scale out for events such as a large scale web cast on a popular topic or to provision instances for a new version of an application.</li>
<li>Partially full hypervisors where the last remaining cores are not being requested (<a href="http://openstack-in-production.blogspot.ch/2014/07/openstack-plays-tetris-stacking-and.html">the Tetris problem</a>).</li>
<li>Compute servers at the end of their lifetime which are used to the full before being removed from the computer centre to make room for new deliveries which are more efficient and in warranty.</li>
</ul>
The characteristics of this workload is that it should be possible to stop an<br />
instance within a short time (a few minutes) compared to a traditional physics job.<br />
<h2 style="text-align: left;">
<b>Resource Management In Openstack</b> </h2>
Operators use project quotas for ensuring the fair sharing of their infrastructure. The problem with this, is that quotas pose as hard limits.This<br />
leads to actually dedicating resources for workloads even if they are not used<br />
all the time or to situations where resources are not available even though<br />
there is quota still to use.<br />
<br />
At the same time, the demand for cloud resources is increasing rapidly. Since<br />
there is no cloud with infinite capabilities, operators need a way to optimize<br />
the resource utilization before proceeding to the expansion of their infrastructure. <br />
<br />
Resources in idle state can occur, showing lower cloud utilization than the full<br />
potential of the acquired equipment while the users’ requirements are growing.<br />
<br />
The concept of Preemptible Instances can be the solution to this problem. These<br />
type of servers can be spawned on top of the project's quota, making use of the<br />
underutilised capabilities. When the resources are requested by tasks with<br />
higher priority (such as approved quota), the preemptible instances are<br />
terminated to make space for the new VM.<br />
<h2 style="text-align: left;">
<b>Preemptible Instances with Openstack</b></h2>
Supporting preemptible instances, would mirror the AWS Spot Market and the<br />
Google Preemptible Instances. There are multiple things to be addressed here as<br />
part of an implementation with OpenStack, but the most important can be reduced to these:<br />
<ol start="1" style="text-align: left;">
<li>Tagging Servers as Preemptible</li>
</ol>
In order to be able to distinguish between preemptible and non-preemptible<br />
servers, there is the need to tag the instances at creation time. This property<br />
should be immutable for the lifetime of the servers.<br />
<ol start="2" style="text-align: left;">
<li>Who gets to use preemptible instances</li>
</ol>
There is also the need to limit which user/project is allowed to use preemptible<br />
instances. An operator should be able to choose which users are allowed to spawn this type of VMs.<br />
<ol start="3" style="text-align: left;">
<li>Selecting servers to be terminated</li>
</ol>
Considering that the preemptible instances can be scattered across the different cells/availability zones/aggregates, there has to be “someone” able to find the existing instances, decide the way to free up the requested resources according to the operator’s needs and, finally, terminate the appropriate VMs.<br />
<ol start="4" style="text-align: left;">
<li>Quota on top of project’s quota</li>
</ol>
<div style="text-align: left;">
In order to avoid possible misuse, there could to be a way to control the amount of preemptible resources that each user/project can use. This means that apart from the quota for the standard resource classes, there could be a way to enforce quotas on the preemptible resources too.<br />
<h2 style="text-align: left;">
<b>OPIE : IFCA and Indigo Dataclouds</b></h2>
</div>
<div style="text-align: left;">
In 2014, there were the first investigations into approaches by Alvaro Lopez<br />
from IFCA (<a href="https://blueprints.launchpad.net/nova/+spec/preemptible-instances">https://blueprints.launchpad.net/nova/+spec/preemptible-instances</a>).<br />
As part of the <a href="https://www.indigo-datacloud.eu/">EU Indigo Datacloud</a> project, this led to the development of the<br />
OpenStack Pre-Emptible Instances package (<a href="https://github.com/indigo-dc/opie">https://github.com/indigo-dc/opie</a>).<br />
This was written up in a paper to Journal of Physics: Conference Series<br />
(<a href="http://iopscience.iop.org/article/10.1088/1742-6596/898/9/092010/pdf">http://iopscience.iop.org/article/10.1088/1742-6596/898/9/092010/pdf</a>) and<br />
presented at the OpenStack summit (<a href="https://www.youtube.com/watch?v=eo5tQ1s9ZxM">https://www.youtube.com/watch?v=eo5tQ1s9ZxM</a>)<br />
<h2 style="text-align: left;">
<b>Prototype Reaper Service</b></h2>
</div>
<div style="text-align: left;">
At the OpenStack Forum during a recent OpenStack summit, a detailed discussion took place on how spot instances could be implemented without significant changes to Nova. The ideas were then followed up with the <a href="https://wiki.openstack.org/wiki/Scientific_SIG">OpenStack Scientific Special Interest Group</a>.<br />
<br />
Trying to address the different aspects of the problem, we are currently<br />
prototyping a “Reaper” service. This service acts as an orchestrator for<br />
preemptible instances. It’s sole purpose is to decide the way to free up the<br />
preemptible resources when they are requested for another task.<br />
<br />
The reason for implementing this prototype, is mainly to help us identify<br />
possible changes that are needed in Nova codebase to support Preemptible<br />
Instances.<br />
<br />
More on this WIP can be found here: </div>
<div style="text-align: left;">
<a href="https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype">https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype</a> </div>
<div style="text-align: left;">
<h2 style="text-align: left;">
<b>Summary</b></h2>
The concept of Preemptible Instances gives operators the ability to provide a<br />
more "elastic" capacity. At the same time, it enables the handling of increased<br />
demand for resources, with the same infrastructure, by maximizing the cloud<br />
utilization.<br />
<br />
This type of servers is perfect for tasks/apps that can be terminated at any<br />
time, enabling the users to take advantage of extra cpu power on demand without the fixed limits that quotas enforce.<br />
<br />
Finally, here in CERN, there is an ongoing effort to provide a prototype<br />
orchestrator for Preemptible Servers with Openstack, in order to pinpoint the<br />
changes needed in Nova to support this feature optimally. This could also be<br />
available in future for other OpenStack clouds in use by CERN such as the<br />
T-Systems Open Telekom Cloud through the Helix Nebula Open Science Cloud<br />
project.<br />
<h2 style="text-align: left;">
Contributors</h2>
</div>
<ul style="text-align: left;">
<li>Theodoros Tsioutsias (CERN openlab fellow working on <a href="https://openlab.cern/project/openstack-clouds">Huawei collaboration</a>)</li>
<li>Spyridon Trigazis (CERN)</li>
<li>Belmiro Moreira (CERN)</li>
</ul>
<div style="text-align: left;">
<h2 style="text-align: left;">
References</h2>
</div>
<ul style="text-align: left;">
<li>CERN Huawei openlab collaboration at <span style="color: #0000ee;"><u>https://openlab.cern/members/huawei </u></span>and <a href="https://openlab.cern/project/openstack-clouds">https://openlab.cern/project/openstack-clouds</a></li>
<li>Helix Nebula Science Cloud project at <a href="http://www.helix-nebula.eu/">http://www.helix-nebula.eu</a> </li>
<li>Indigo Datacloud at <a href="https://www.indigo-datacloud.eu/">https://www.indigo-datacloud.eu/</a></li>
<li>CERN SKA collaboration plans at <a href="https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska">https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska</a></li>
<li>OpenStack Scientific Special Interest Group at <a href="https://wiki.openstack.org/wiki/Scientific_SIG">https://wiki.openstack.org/wiki/Scientific_SIG</a></li>
<li>Nova patch to support Reaper - <a href="https://review.openstack.org/547450">https://review.openstack.org/547450</a></li>
</ul>
</div>
Theodoros Tsioutsiashttp://www.blogger.com/profile/01144068404522382207noreply@blogger.com16tag:blogger.com,1999:blog-6032896665180559.post-30276836322583582472018-01-30T05:31:00.000-08:002018-01-30T05:31:33.846-08:00Keep calm and reboot: Patching recent exploits in a production cloudAt CERN, we have around 8,500 hypervisors running 36,000 guest virtual machines. These provide the compute resources for both the laboratory's physics program but also for the organisation's administrative operations such as paying bills and reserving rooms at the hostel. These resources are spread over many different server configurations, some of them over 5 years old.<br />
<br />
With the accelerator stopping over the CERN annual closure until <a href="https://lhc-commissioning.web.cern.ch/lhc-commissioning/schedule/2018-LHC-schedule_v0.3.pdf">mid March</a>, this is a good period to be planning reconfiguration of compute resources such as the migration of our central batch system which schedules the jobs across the central compute resources to a new system based on HTCondor. The compute resources are heavily used but there is more flexibility to drain some parts in the quieter periods of the year when there is not 10PB/month coming from the detectors. However, this year we have had an unexpected additional task to deploy the fixes for the Meltdown and Spectre exploits across the centre.<br />
<br />
The CERN environment is based on Scientific Linux CERN 6 and CentOS 7. The hypervisors are now entirely CentOS 7 based with guests of a variety of operating systems including Windows flavors and <a href="https://cernvm.cern.ch/">CERNVM</a>. The campaign to upgrade involved a number of steps<br />
<ul>
<li>Assess the security risk</li>
<li>Evaluate the performance impact</li>
<li>Test the upgrade procedure and stability</li>
<li>Plan the upgrade campaign</li>
<li>Communicate with the users</li>
<li>Execute the campaign</li>
</ul>
<h2>
Security Risk</h2>
<div>
The CERN environment consists of a mixture of different services, with thousands of projects on the cloud, distributed across two data centres in Geneva and Budapest. </div>
<div>
<br /></div>
<div>
Two major risks were identified</div>
<div>
<ul>
<li>Services which provided the ability for end users to run their own programs along with others sharing the same kernel. Examples of this are the public login services and batch farms. Public login services provide an interactive Linux environment for physicists to log into from around the world, prepare papers, develop and debug applications and submit jobs to the central batch farms. The batch farms themselves provide 1000s of worker nodes processing the data from CERN experiments by farming event after event to free compute resources. Both of these environments are multi-user and allow end users to compile their own programs and thus were rated as high risk for the Meltdown exploit.</li>
<li>The hypervisors provide support for a variety of different types of virtual machines. Different areas of the cloud provide access to different network domains or to compute optimised configurations. Many of these hypervisors will have VMs owned by different end users and therefore can be exposed to the Spectre exploits, even if the performance is such that exploiting the problem would take significant computing time.</li>
</ul>
The remaining VMs are for dedicated services without access for end user applications or dedicated bare metal servers for I/O intensive applications such as databases and disk or tape servers.</div>
<div>
<br /></div>
<div>
There are a variety of different hypervisor configurations which we split down by processor type (in view of the Spectre microcode patches). Each of these needs independent performance and stability checks.</div>
<div>
<br /></div>
<div>
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
<o:PixelsPerInch>96</o:PixelsPerInch>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-GB</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="382">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 9"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:1;
mso-generic-font-family:roman;
mso-font-format:other;
mso-font-pitch:variable;
mso-font-signature:0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073786111 1 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-fareast-language:EN-US;}
</style>
<![endif]-->
<!--StartFragment-->
<br />
<table border="0" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: whitesmoke; border-collapse: collapse; margin-left: 1.5pt; mso-yfti-tbllook: 1184; width: 473px;">
<tbody>
<tr>
<td style="border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;" width="20%"><div align="center" class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt; text-align: center;">
<b><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">Microcode<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;" width="20%"><div align="center" class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt; text-align: center;">
<b><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">Assessment<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;" width="10%"><div align="center" class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt; text-align: center;">
<b><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">#HVs<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div align="center" class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt; text-align: center;">
<b><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">Processor name(s)<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">06-3f-02<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #14892c; font-family: "arial" , sans-serif; font-size: 10.5pt;">covered</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #14892c; font-family: "arial" , sans-serif; font-size: 10.5pt;">3332</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">E5-2630 v3 @ 2.40GHz,E5-2640 v3 @ 2.60GHz<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">06-4f-01<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #14892c; font-family: "arial" , sans-serif; font-size: 10.5pt;">covered</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #14892c; font-family: "arial" , sans-serif; font-size: 10.5pt;">2460</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">E5-2630 v4 @ 2.20GHz, E5-2650 v4 @ 2.20GHz<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">06-3e-04<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #f79232; font-family: "arial" , sans-serif; font-size: 10.5pt;">hopefully</span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #f79232; font-family: "arial" , sans-serif; font-size: 10.5pt;">1706</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">E5-2650 v2 @ 2.60GHz<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">??<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #d04437; font-family: "arial" , sans-serif; font-size: 10.5pt;">unclear</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: red; font-family: "arial" , sans-serif; font-size: 10.5pt;">427</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">CPU family: 21 Model: 1 Model name: AMD Opteron(TM) Processor 6276
Stepping: 2<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">06-2d-07</span></div>
</td><td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;">unclear</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #d04437; font-family: "arial" , sans-serif; font-size: 10.5pt;">333</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">E5-2630L 0 @ 2.00GHz, E5-2650 0 @ 2.00GHz<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #CCCCCC 1.0pt; mso-border-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">06-2c-02<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #d04437; font-family: "arial" , sans-serif; font-size: 10.5pt;">unlikely</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt;"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: red; font-family: "arial" , sans-serif; font-size: 10.5pt;">168</span><span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #CCCCCC 1.0pt; border-left: none; border-right: solid #CCCCCC 1.0pt; border-top: none; mso-border-alt: solid #CCCCCC .75pt; mso-border-left-alt: solid #CCCCCC .75pt; mso-border-top-alt: solid #CCCCCC .75pt; padding: 2.25pt 3.0pt 2.25pt 3.0pt; width: 307.4pt;" width="307"><div class="MsoNormal" style="margin-bottom: 3.75pt; margin-top: 3.75pt;">
<span style="color: #333333; font-family: "arial" , sans-serif; font-size: 10.5pt;">E5645 @ 2.40GHz, L5640 @ 2.27GHz, X5660 @ 2.80GHz<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<!--EndFragment--></div>
<div>
<br /></div>
<div>
These risks were explained by the CERN security team to the end users in their regular <a href="https://security.web.cern.ch/security/advisories/spectre-meltdown/spectre-meltdown.shtml">blogs</a>.</div>
<h2>
Evaluating the performance impact</h2>
<div>
The High Energy Physics community uses a suite called <a href="http://w3.hepix.org/benchmarking.html">HEPSPEC06</a> to benchmark compute resources. These are synthetic programs based on the C++ components of SPEC CPU2006 which match the instruction mix of the typical physics programs. With this benchmark, we have started to re-benchmark (the majority of) the CPU models we have in the data centres, both on the physical hosts and on the guests. The measured performance loss across all architectures tested so far is about 2.5% in HEPSPEC06 (a number also confirmed by by one of the LHC experiments using their real workloads) with a few cases approaching 7%. So for our physics codes, the effect of patching seems measurable, but much smaller than many expected. </div>
<h2>
Test the upgrade procedure and stability</h2>
<div>
With our environment based on CentOS and Scientific Linux, the deployment of the updates for Meltdown and Spectre were dependent on the upstream availability of the patches. These could be broken down into several parts</div>
<div>
<ul>
<li>Firmware for the processors - the microcode_ctl packages provide additional patches to protect against some parts of Spectre. This package proved very dynamic as new processor firmware was being added on a regular basis and it was not always clear when this needed to be applied, the package version would increase but it was not always that this included an update for the particular hardware type. Following through the <a href="https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File?product=122139">Intel release notes</a>, there were combinations such as "<span style="font-family: "courier new" , "courier" , monospace;">HSX C0(06-3f-02:6f) 3a->3b</span>" which explains that the processor description 06-3f-02:6f is upgraded from release 0x3a to 0x3b. The fields are the CPU family, model and stepping from <span style="font-family: "courier new" , "courier" , monospace;">/proc/cpuinfo</span> and the firmware level can be found at <span style="font-family: "courier new" , "courier" , monospace;">/sys/devices/system/cpu/cpu0/microcode/version</span>. A simple script (<a href="https://security.web.cern.ch/security/advisories/spectre-meltdown/spectre-cpu-microcode-checker.sh">spectre-cpu-microcode-checker.sh</a>) was made available to the end users so they could check their systems and this was also used by the administrators to validate the central IT services.</li>
<li>For the operating system, we used a second script (<a href="https://security.web.cern.ch/security/advisories/spectre-meltdown/spectre-meltdown-checker.sh">spectre-meltdown-checker.sh</a>) which was derived from the upstream github code at <a href="https://github.com/speed47/spectre-meltdown-checker">https://github.com/speed47/spectre-meltdown-checker</a>. The team maintaining this package were very responsive incorporating our patches so that other sites could benefit from the combined analysis.</li>
</ul>
</div>
<div>
<h2>
Communication with the users</h2>
</div>
<div>
For the cloud, there are several resource consumers.</div>
<div>
<ul>
<li>IT service administrators who provide higher level functions on top of the CERN cloud. Examples include file transfer services, information systems, web frameworks and experiment workload management systems. While some are in the IT department, others are representatives of their experiments or supporters for online control systems such as those used to manage the accelerator infrastructure.</li>
<li>End users consume cloud resources by asking for virtual machines and using them as personal working environments. Typical cases would be a MacOS user who needs a Windows desktop where they would create a Windows VM and use protocols such as RDP to access it when required.</li>
</ul>
<div>
The communication approach was as follows:</div>
</div>
<div>
<ul>
<li>A meeting was held to discuss the risks of exploits, the status of the operating systems and the plan for deployment across the production facilities. With a Q&A session, the major concerns raised were around potential impact on performance and tuning options. </li>
<li>An e-mail was sent to all owners of virtual machine resources informing them of the upcoming interventions.</li>
<li>CERN management was informed of the risks and the plan for deployment.</li>
</ul>
<div>
CERN uses ServiceNow to provide a service desk for tickets and a <a href="http://cern.ch/itssb">status board of interventions</a> and incidents. A single entry was used to communicate the current plans and status so that all cloud consumers could go to a single place for the latest information.</div>
</div>
<h2>
Execute the campaign</h2>
<div>
With the accelerator starting up again in March and the risk of the exploits, the approach taken was to complete the upgrades to the infrastructure in January, leaving February to find any residual problems and resolve them. As the handling of the compute/batch part of the infrastructure was relatively straight forward (with only one service on top), we will focus in the following on the more delicate part of hypervisors running services supporting several thousand users in their daily work.<br />
<br />
The layout of our infrastructure with its availability zones (AVZs) determined the overall structure and timeline of the upgrade. With effectively four AVZs in our data centre in Geneva and two AVZs for our remote resources in Budapest, we scheduled the upgrade for the services part of the resources over four days.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-0Rqa9VE-7Q8/Wm7v32tTeyI/AAAAAAAAOUQ/Z2BL0G1iarIBtr_wryEhk-PREX-jYg7IQCLcBGAs/s1600/Screen%2BShot%2B2018-01-29%2Bat%2B10.56.23.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="516" data-original-width="1052" height="312" src="https://2.bp.blogspot.com/-0Rqa9VE-7Q8/Wm7v32tTeyI/AAAAAAAAOUQ/Z2BL0G1iarIBtr_wryEhk-PREX-jYg7IQCLcBGAs/s640/Screen%2BShot%2B2018-01-29%2Bat%2B10.56.23.png" width="640" /></a></div>
The main zones in Geneva were done one per day, with a break after the first one (GVA-A) in case there were unexpected difficulties to handle on the infrastructure or on the application side. The remaining zones were scheduled on consecutive days (GVA-B and GVA-C), the smaller ones (critical, WIG-A, WIG-B) in sequential order on the last day. This way we upgraded around 400 hosts with 4,000 guests per day.<br />
<br />
Within each zone, hypervisors were divided into 'reboot groups' which were restarted and checked before the next group was handled. These groups were determined by the OpenStack cells underlying the corresponding AVZs. Since some services required to limit the window of service downtime, their hosting servers were moved to the special Group 1, the only one for which we could give a precise start time.<br />
<br />
For each group several steps were performed:<br />
<ul>
<li>install all relevant packages</li>
<li>check the next kernel is the desired one</li>
<li>reset the BMC (needed for some specific hardware to prevent boot problems)</li>
<li>log the nova and ping state of all guests</li>
<li>stop all alarming </li>
<li>stop nova</li>
<li>shut down all instances via virsh</li>
<li>reboot the hosts</li>
<li>... wait ... then fix hosts which did not come back</li>
<li>check running kernel and vulnerability status on the rebooted hosts</li>
<li>check and fix potential issues with the guests</li>
</ul>
Shutting down virtual machines via 'virsh', rather than the OpenStack APIs, was chosen to speed up the overall process -- even if this required to switch off nova-compute on the hosts as well (to keep nova in a consistent state). An alternative to issuing 'virsh' commands directly would be to configure 'libvirt-guests', especially in the context of the question whether guests should be shut down and rebooted (which we did during this campaign) or paused/resumed. This is an option we'll have a look at to prepare for similar campaigns in the future.<br />
<br />
As some of the hypervisors in the cloud had very long uptimes and this was the first time we systematically rebooted the whole infrastructure since the service went to full production about five years ago, we were not quite sure what kind issues to expect -- and in particular at which scale. To our relief, the problems encountered on the hosts hit less than 1% of the servers and included (in descending order of appearance)<br />
<ul>
<li>hosts stuck in shutdown (solved by IPMI reset)</li>
<li>libvirtd stuck after reboot (solved by another reboot)</li>
<li>hosts without network connectivity (solved by another reboot)</li>
<li>hosts stuck in grub during boot (solved by reinstalling grub) </li>
</ul>
</div>
<div>
On the guest side, virtual machines were mostly ok when the underlying hypervisor was ok as well.<br />
A few additional cases included<br />
<ul>
<li>incomplete kernel upgrades, so the root partition could not be found (solved by booting back into an older kernel and reinstall the desired kernel)</li>
<li>file system issues (solved by running file system repairs)</li>
</ul>
<div>
So, despite initial worries, we hit no major issues when rebooting the whole CERN cloud infrastructure!<br />
<h2>
Conclusions</h2>
<div>
While these kind of security issues do not arrive very often, the key parts of the campaign follow standard steps, namely assessing the risk, planning the update, communicating with the user community, execution and handling incomplete updates.</div>
<div>
<br /></div>
<div>
Using cloud availability zones to schedule the deployment allowed users to easily understand when there would be an impact on their virtual machines and encourages good practise to load balance resources.</div>
<h2>
References</h2>
<div>
<ul>
<li>CERN security <a href="https://security.web.cern.ch/security/advisories/spectre-meltdown/spectre-meltdown.shtml">advisory</a> for Meltdown / Spectre</li>
<li>CERN IT <a href="https://cern.service-now.com/service-portal/ssb.do?area=IT">Status Board</a> </li>
</ul>
</div>
<h2>
Authors</h2>
</div>
</div>
<div>
<ul>
<li>Arne Wiebalck</li>
<li>Jan Van Eldik</li>
<li>Tim Bell</li>
</ul>
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com90Geneva, Switzerland46.2043907 6.143157699999960646.1164762 5.9817961999999607 46.292305199999994 6.3045191999999606tag:blogger.com,1999:blog-6032896665180559.post-83792682444454335322017-08-30T08:26:00.002-07:002017-09-06T00:59:43.792-07:00Scheduled snapshots<div class="MsoNormal">
<div class="MsoNormal">
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">While most of the machines on the CERN cloud are configured using
Puppet with state stored in external databases or file stores, there are a few
machines where this has been difficult, especially for legacy applications. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Doing a regular snapshot of these machines would be a way of
protecting against failure scenarios such as hypervisor failure or disk
corruptions.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">This could always be scripted by the project administrator using
the standard functions in the openstack client but this would also involve
setting up the schedules and the credentials externally to the cloud along with
appropriate skills for the project administrators. Since it is a common
request, the CERN cloud investigated how this could be done as part of the
standard cloud offering.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The approach that we have taken uses the Mistral project to execute
the appropriate workflows at a scheduled time. The CERN cloud is running a
mixture of OpenStack Newton and Ocata but we used the Mistral Pike release in
order to have the latest set of fixes such as in the cron triggers. With the
RDO packages coming out in the same week as the upstream release, this avoided
doing an upgrade later.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Mistral has a set of terms which explain the different parts of a
workflow (</span><a href="https://docs.openstack.org/mistral/latest/terminology"><span style="font-family: "Times",serif; font-size: 13.5pt;">https://docs.openstack.org/mistral/latest/terminology</span></a><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">). <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The approach needed several steps<o:p></o:p></span></div>
<ul type="disc">
<li class="MsoNormal" style="color: black; mso-list: l2 level1 lfo1; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral tasks to define the steps<o:p></o:p></span></li>
<li class="MsoNormal" style="color: black; mso-list: l2 level1 lfo1; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral workflows to provide the order to perform the
steps in<o:p></o:p></span></li>
<li class="MsoNormal" style="color: black; mso-list: l2 level1 lfo1; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral cron triggers to execute the steps on schedule<o:p></o:p></span></li>
</ul>
<div>
<span style="font-family: Times, serif;"><span style="font-size: 18px;"><br /></span></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral Workflows<o:p></o:p></span></b></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;"><br /></span></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The Mistral workflows consist of a set of tasks and a process
which decides which task to execute next based on different branch criteria
such as success of a previous task or the value of some cloud properties.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Workflows can be private to the project, shared or public. By
making these scheduled snapshot workflows public, the cloud administrators can
improve the tasks incrementally and the cloud projects will receive the latest
version of the workflow next time they execute them. With the CERN gitlab based
continuous integration environment, the workflows are centrally maintained and
then pushed to the cloud when the test suites have completed successfully.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The following Mistral workflows were defined<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;"><br /></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 4;">
<b><span style="color: black; font-family: "Times",serif; mso-fareast-font-family: "Times New Roman";">instance_snapshot<o:p></o:p></span></b></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;"><br /></span></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Virtual machines can be snapshotted so that a copy of the virtual
machine is saved and can be used for recovery or cloning in future. The
instance_snapshot workflow performs this operation for both virtual machines
which have been booted from volume or locally.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<table align="left" border="1" cellpadding="0" class="MsoNormalTable" style="margin-left: 6.0pt; margin-right: 6.0pt; mso-cellspacing: 1.5pt; mso-table-anchor-horizontal: page; mso-table-anchor-vertical: paragraph; mso-table-left: 76.8pt; mso-table-lspace: 9.0pt; mso-table-rspace: 9.0pt; mso-table-top: inside; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Parameter<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Description<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Default<o:p></o:p></span></b></div>
</td>
</tr>
<tr style="mso-yfti-irow: 1;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">instance<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name of the instance to
be snapshot<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Mandatory<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 2;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">pattern<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name of the snapshot to
store. The text ={0}= is replaced by the instance name and the text ={1}= is
replaced by the date in the format YYYYMMDDHHMM.<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">{0}_snapshot_{1}<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 3;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">max_snapshots<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The number of snapshots to
keep. Older snapshots are cleaned from the store when new ones are created.<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">0 (i.e. keep all)<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 4;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">wait<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Only complete the workflow
when the steps have been completed and the snapshot is stored in the image
storage<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">false<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 5;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">instance_stop<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Shut the instance down
before snapshotting and boot it up afterwards.<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">false (i.e. do not stop the
instanc)<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 6;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">to_addr_success<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">e-mail address to send the
report if the workflow is successful<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">null (i.e. no mail sent)<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 7; mso-yfti-lastrow: yes;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">to_addr_error<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">e-mail address to send the
report if the workflow failed<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">null (i.e. no mail sent)<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The steps for this workflow are described in the detail in the
YAML/YAQL files at </span><a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows"><span style="font-family: "Times",serif; font-size: 13.5pt;">https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows</span></a><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The operation is very fast with Ceph based boot-from-volumes since
the snapshot is done within Ceph. It can however take up to a minute for
locally booted VMs while the hypervisor is ensuring the complete disk contents
are available. The VM is resumed and the locally booted snapshot is then sent
to Glance in the background.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The high level steps are<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Identify server<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Stop instance if
requested by instance_stop<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">If the VM is
locally booted<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt; mso-fareast-font-family: "Courier New";"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Snapshot the
instance<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt; mso-fareast-font-family: "Courier New";"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Clean up the
oldest image snapshot if over max_snapshots<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">If the VM is
booted from volume<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt; mso-fareast-font-family: "Courier New";"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Snapshot the
volume<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt; mso-fareast-font-family: "Courier New";"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Cleanup oldest
volume snapshot if over max_snapshots<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Start instance
if requested by instance_stop<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">If there is an
error and to_addr_error is set<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt; mso-fareast-font-family: "Courier New";"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Send an e-mail
to to_addr_error<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l3 level1 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">If there is no
error and to_addr_success is set<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt; mso-fareast-font-family: "Courier New";"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Send an e-mail
to to_addr_success<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l3 level2 lfo6; text-indent: -18.0pt;">
<br /></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 4;">
<b><span style="color: black; font-family: "Times",serif; mso-fareast-font-family: "Times New Roman";">restore_clone_snapshot<o:p></o:p></span></b></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">For applications which are not highly available, a common
configuration is using a LanDB alias to a particular VM. In the event of a
failure, the VM can be cloned from a snapshot and the LanDB alias updated to
reflect the new endpoint location for the service. This workflow will create a
volume if the source instance is booted from volume. The workflow is called
restore_clone_snapshot.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">The source instance needs to be still defined since information
such as the properties, flavor and availability zone are not included in the
snapshot and these are propagated by default.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<table align="left" border="1" cellpadding="0" class="MsoNormalTable" style="margin-left: 6.0pt; margin-right: 6.0pt; mso-cellspacing: 1.5pt; mso-table-anchor-horizontal: page; mso-table-anchor-vertical: paragraph; mso-table-left: 76.8pt; mso-table-lspace: 9.0pt; mso-table-rspace: 9.0pt; mso-table-top: inside; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Parameter<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Description<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Default<o:p></o:p></span></b></div>
</td>
</tr>
<tr style="mso-yfti-irow: 1;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">instance<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="color: black; font-family: "Times",serif; font-size: 8.0pt;">The name of the
instance from which the snapshot will be cloned</span><span style="font-family: "Times",serif; font-size: 8.0pt;"><o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Mandatory<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 2;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Date<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The date of the snapshot to
clone (either YYYYMMDD or YYYYMMDDHHMM)<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Mandatory<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 3;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">pattern<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name of the snapshot to
clone. The text ={0}= is replaced by the instance name and the text ={1}= is
replaced by the date.<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">{0}_snapshot_{1}<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 4;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">clone_name<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name of the new
instance to be created<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Mandatory<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 5;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">avz_name<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The availability zone to
create the clone in. <o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Same as the source instance<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 6;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">flavor<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The flavour for the cloned
instance<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Same as the source instance<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 7;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">meta<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The properties to copy to
the new instance<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">All properties are copied
from the source<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559#_ftn1" name="_ftnref1" style="mso-footnote-id: ftn1;" title=""><span class="MsoFootnoteReference"><span style="mso-special-character: footnote;"><!--[if !supportFootnotes]--><span class="MsoFootnoteReference"><span style="font-family: "Times",serif; font-size: 8.0pt; mso-ansi-language: EN-GB; mso-bidi-font-family: "Times New Roman"; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-GB; mso-fareast-theme-font: minor-latin;">[1]</span></span><!--[endif]--></span></span></a><o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 8;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">wait<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Only complete the workflow
when the steps have been completed and the cloned VM is running<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">false<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 9;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">to_addr_success<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">e-mail address to send the
report if the workflow is successful<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">null (i.e. no mail sent)<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 10; mso-yfti-lastrow: yes;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">to_addr_error<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">e-mail address to send the
report if the workflow failed<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">null (i.e. no mail sent)<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Thus, cloning the machine timbfvlinux143 to timbfvclone143
requires running the workflow with the parameters<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Courier New",serif; font-size: 13.5pt;">{“instance”: “timbfvlinux143”, “clone_name”: “timbfvclone143”,
“date”: “20170830” }<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">This results in<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l4 level1 lfo5; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">A new volume
created from the snapshot timbfvlinux143_snapshot_20170830<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l4 level1 lfo5; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">A new VM is
created called timbfvclone143 booted from the new volume<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">An instance clone can be run for VMs which are booted from volume
even when the hypervisor is not running. A machine can then be recovered from
it's current state using the procedure<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l5 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Instance
snapshot of original machine<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l5 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">Instance clone
from that snapshot (using today's date)<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l5 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="color: black; font-family: Symbol; font-size: 13.5pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">If DNS aliases
are used, the alias can then be updated to point to the new instance name<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">For Linux guests, the rename of the hostname to the clone name occurs
as the machine is booted. In the CERN environment, this took a few minutes to
create the new virtual machine and then up to 10 minutes to wait for the DNS
refresh.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">For Windows guests, it may be necessary to refresh the Active
Directory information given the change of hostname.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 4;">
<b><span style="color: black; font-family: "Times",serif; mso-fareast-font-family: "Times New Roman";">restore_inplace_snapshot<o:p></o:p></span></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">In the event of an issue such as a bad upgrade, the administrator
may wish to roll back to the last snapshot. This can be done using the
restore_inplace_snapshot workflow.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">This operation works for locally booted machines, maintains the IP
and MAC address but cannot be used if the hypervisor is down. It does not
currently work for boot from volume until the revert to snapshot (available in
Pike from </span><a href="https://specs.openstack.org/openstack/cinder-specs/specs/pike/cinder-volume-revert-by-snapshot.html"><span style="font-family: "Times",serif; font-size: 13.5pt;">https://specs.openstack.org/openstack/cinder-specs/specs/pike/cinder-volume-revert-by-snapshot.html</span></a><span style="color: black; font-family: "Times",serif; font-size: 13.5pt;">) is in
production.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<table align="left" border="1" cellpadding="0" class="MsoNormalTable" style="margin-left: 6.0pt; margin-right: 6.0pt; mso-cellspacing: 1.5pt; mso-table-anchor-horizontal: page; mso-table-anchor-vertical: paragraph; mso-table-left: 76.8pt; mso-table-lspace: 9.0pt; mso-table-rspace: 9.0pt; mso-table-top: inside; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Parameter<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Description<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Default<o:p></o:p></span></b></div>
</td>
</tr>
<tr style="mso-yfti-irow: 1;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">instance<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="color: black; font-family: "Times",serif; font-size: 8.0pt;">The name of the
instance from which the snapshot will be replaced</span><span style="font-family: "Times",serif; font-size: 8.0pt;"><o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Mandatory<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 2;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">date<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The date of the snapshot to
replace from (either YYYYMMDD or YYYYMMDDHHMM)<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Mandatory<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 3;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">pattern<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name of the snapshot to
replace from. The text ={0}= is replaced by the instance name and the text
={1}= is replaced by the date.<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">{0}_snapshot_{1}<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 4;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">wait<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Only complete the workflow
when the steps have been completed and the replaced VM is running<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">false<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 5;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">to_addr_success<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">e-mail address to send the
report if the workflow is successful<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">null (i.e. no mail sent)<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 6; mso-yfti-lastrow: yes;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">to_addr_error<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">e-mail address to send the
report if the workflow failed<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">null (i.e. no mail sent)<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral Cron Triggers<o:p></o:p></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral has another nice
feature where it is able to run a workflow at regular intervals. Compared to
standard Unix cron, the Mistral cron triggers use Keystone trusts to save the
user token when the trigger is enabled. Thus, the execution is able to run
without needing the credentials such as a password or valid Kerberos token.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">The steps are as follows
to create a cron trigger via Horizon or the CLI.<o:p></o:p></span></div>
<table align="left" border="1" cellpadding="0" class="MsoNormalTable" style="margin-left: 6.0pt; margin-right: 6.0pt; mso-cellspacing: 1.5pt; mso-table-anchor-horizontal: page; mso-table-anchor-vertical: paragraph; mso-table-left: 76.8pt; mso-table-lspace: 9.0pt; mso-table-rspace: 9.0pt; mso-table-top: inside; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Parameter<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Description<o:p></o:p></span></b></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div align="center" class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly; text-align: center;">
<b><span style="font-family: "Times",serif; font-size: 8.0pt;">Example<o:p></o:p></span></b></div>
</td>
</tr>
<tr style="mso-yfti-irow: 1;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Name<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name of the cron
trigger<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Nightly Snapshot<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 2;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Workflow ID<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">The name or UUID of the
workflow<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">instance_snapshot<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 3;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Params<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">A JSON dictionary of the
parameters<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">{“instance”:
“timbfvlinux143”, “max_snapshots”: 5, “to_addr_error”: “theadmin@cern.ch”}<o:p></o:p></span></div>
</td>
</tr>
<tr style="mso-yfti-irow: 4; mso-yfti-lastrow: yes;">
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">Pattern<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">A cron schedule pattern
according to http://en.wikipedia.org/wiki/Cron<o:p></o:p></span></div>
</td>
<td style="padding: .75pt .75pt .75pt .75pt;">
<div class="MsoNormal" style="mso-element-anchor-horizontal: page; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: 76.85pt; mso-element-top: inside; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;">
<span style="font-family: "Times",serif; font-size: 8.0pt;">* 5 * * * (i.e. run daily
at 5a.m.)<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<br /></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">This will then execute
the instance snapshot at 5a.m. sending a mail to </span><a href="mailto:theadmin@cern.ch"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">theadmin@cern.ch</span></a><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"> in the event of a failure of the snapshot. 5 past copies will be
kept.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral Executions<o:p></o:p></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">When Mistral runs a
workflow, it provides details of the steps executed, the timestamps for start
and end along with the results. Each step can be inspected individually as part
of debugging and root cause analysis in the event of failures.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">The Horizon interface gives
an easy interface for selecting the failing tasks. There may be tasks reported
as ‘error’ but these steps can then have subsequent actions which succeed so an
error step may be a normal part of a successful task execution such as using a
default if no value can be found.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-7mZR0GW4Cdw/WawRn3VMQPI/AAAAAAAAUKw/yCHrkHq2yncJ9jljJBFM3IzauqZ_TIAwwCPcBGAYYCw/s1600/mistral%2Btasks.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="648" data-original-width="1600" height="257" src="https://4.bp.blogspot.com/-7mZR0GW4Cdw/WawRn3VMQPI/AAAAAAAAUKw/yCHrkHq2yncJ9jljJBFM3IzauqZ_TIAwwCPcBGAYYCw/s640/mistral%2Btasks.png" width="640" /></a></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman"; mso-no-proof: yes;"><!--[if gte vml 1]><v:shapetype
id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t"
path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f">
<v:stroke joinstyle="miter"/>
<v:formulas>
<v:f eqn="if lineDrawn pixelLineWidth 0"/>
<v:f eqn="sum @0 1 0"/>
<v:f eqn="sum 0 0 @1"/>
<v:f eqn="prod @2 1 2"/>
<v:f eqn="prod @3 21600 pixelWidth"/>
<v:f eqn="prod @3 21600 pixelHeight"/>
<v:f eqn="sum @0 0 1"/>
<v:f eqn="prod @6 1 2"/>
<v:f eqn="prod @7 21600 pixelWidth"/>
<v:f eqn="sum @8 21600 0"/>
<v:f eqn="prod @7 21600 pixelHeight"/>
<v:f eqn="sum @10 21600 0"/>
</v:formulas>
<v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/>
<o:lock v:ext="edit" aspectratio="t"/>
</v:shapetype><v:shape id="Picture_x0020_2" o:spid="_x0000_i1025" type="#_x0000_t75"
style='width:451pt;height:183pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:////Users/timbell/Library/Group%20Containers/UBF8T346G9.Office/msoclip1/01/clip_image001.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--></span><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">References<o:p></o:p></span></b></div>
<ul type="disc">
<li class="MsoNormal" style="color: black; mso-list: l1 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Mistral
documentation <a href="https://docs.openstack.org/mistral/latest/">https://docs.openstack.org/mistral/latest/</a><o:p></o:p></span></li>
<li class="MsoNormal" style="color: black; mso-list: l1 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">YAQLuator for testing YAQL expressions - </span><span style="color: windowtext;"><a href="http://yaqluator.com/"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">http://yaqluator.com/</span></a></span><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal" style="color: black; mso-list: l1 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">CERN Mistral Workflows source code - </span><span style="color: windowtext;"><a href="https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows</span></a></span><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal" style="color: black; mso-list: l1 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Computing at CERN - </span><span style="color: windowtext;"><a href="http://home.cern/about/computing"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">http://home.cern/about/computing</span></a></span><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal" style="color: black; mso-list: l1 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">CERN IT department – </span><span style="color: windowtext;"><a href="http://cern.ch/it"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">http://cern.ch/it</span></a></span><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
</ul>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><br /></span></b></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; mso-outline-level: 3;">
<b><span style="color: black; font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Credits<o:p></o:p></span></b></div>
<ul type="disc">
<li class="MsoNormal" style="color: black; mso-list: l0 level1 lfo3; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">Jose Castro Leon from the CERN IT cloud team did the
implementation of the Mistral project and the workflows described.<o:p></o:p></span></li>
</ul>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">
<br /></div>
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
<o:PixelsPerInch>96</o:PixelsPerInch>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves>false</w:TrackMoves>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-GB</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="382">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 9"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:fixed;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:Times;
panose-1:2 0 5 0 0 0 0 0 0 0;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;
mso-font-charset:2;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:0 268435456 0 0 -2147483648 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:-536870145 1107305727 0 0 415 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073786111 1 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoFootnoteText, li.MsoFootnoteText, div.MsoFootnoteText
{mso-style-priority:99;
mso-style-link:"Footnote Text Char";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
span.MsoFootnoteReference
{mso-style-priority:99;
vertical-align:super;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
mso-themecolor:hyperlink;
text-decoration:underline;
text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-noshow:yes;
mso-style-priority:99;
color:#954F72;
mso-themecolor:followedhyperlink;
text-decoration:underline;
text-underline:single;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
span.FootnoteTextChar
{mso-style-name:"Footnote Text Char";
mso-style-priority:99;
mso-style-unhide:no;
mso-style-locked:yes;
mso-style-link:"Footnote Text";
font-family:"Times New Roman",serif;
mso-ascii-font-family:"Times New Roman";
mso-hansi-font-family:"Times New Roman";
mso-bidi-font-family:"Times New Roman";
mso-fareast-language:EN-GB;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
/* Page Definitions */
@page
{mso-footnote-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") fs;
mso-footnote-continuation-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") fcs;
mso-endnote-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") es;
mso-endnote-continuation-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") ecs;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:35.4pt;
mso-footer-margin:35.4pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:431903523;
mso-list-template-ids:2111084500;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:703864863;
mso-list-template-ids:-178648644;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:720442611;
mso-list-template-ids:-911446556;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3
{mso-list-id:1038092977;
mso-list-type:hybrid;
mso-list-template-ids:-1451833032 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4
{mso-list-id:1076441205;
mso-list-type:hybrid;
mso-list-template-ids:1926151842 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l4:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l4:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l4:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l4:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l5
{mso-list-id:2004820809;
mso-list-type:hybrid;
mso-list-template-ids:-1403886720 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l5:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l5:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l5:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l5:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l5:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l5:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l5:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l5:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l5:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-fareast-language:EN-US;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--><br />
<div style="mso-element: footnote-list;">
<!--[if !supportFootnotes]--><br clear="all" />
<hr align="left" size="1" width="33%" />
<!--[endif]-->
<div id="ftn1" style="mso-element: footnote;">
<div class="MsoFootnoteText">
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559#_ftnref1" name="_ftn1" style="mso-footnote-id: ftn1;" title=""><span class="MsoFootnoteReference"><span style="mso-special-character: footnote;"><!--[if !supportFootnotes]--><span class="MsoFootnoteReference"><span style="font-family: "Times New Roman",serif; font-size: 12.0pt; mso-ansi-language: EN-GB; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-GB; mso-fareast-theme-font: minor-latin;">[1]</span></span><!--[endif]--></span></span></a> <span lang="EN-US" style="mso-ansi-language: EN-US;">Except for a CERN specific one
called landb-alias for a DNS alias<o:p></o:p></span></div>
</div>
</div>
</div>
</div>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:fixed;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:Times;
panose-1:2 0 5 0 0 0 0 0 0 0;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;
mso-font-charset:2;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:0 268435456 0 0 -2147483648 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:-536870145 1107305727 0 0 415 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073786111 1 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoFootnoteText, li.MsoFootnoteText, div.MsoFootnoteText
{mso-style-priority:99;
mso-style-link:"Footnote Text Char";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
span.MsoFootnoteReference
{mso-style-priority:99;
vertical-align:super;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
mso-themecolor:hyperlink;
text-decoration:underline;
text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-noshow:yes;
mso-style-priority:99;
color:#954F72;
mso-themecolor:followedhyperlink;
text-decoration:underline;
text-underline:single;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
span.FootnoteTextChar
{mso-style-name:"Footnote Text Char";
mso-style-priority:99;
mso-style-unhide:no;
mso-style-locked:yes;
mso-style-link:"Footnote Text";
font-family:"Times New Roman",serif;
mso-ascii-font-family:"Times New Roman";
mso-hansi-font-family:"Times New Roman";
mso-bidi-font-family:"Times New Roman";
mso-fareast-language:EN-GB;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
/* Page Definitions */
@page
{mso-footnote-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") fs;
mso-footnote-continuation-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") fcs;
mso-endnote-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") es;
mso-endnote-continuation-separator:url("/Users/timbell/Library/Group Containers/UBF8T346G9.Office/msoclip1/01/clip_header.htm") ecs;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:35.4pt;
mso-footer-margin:35.4pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:431903523;
mso-list-template-ids:2111084500;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:703864863;
mso-list-template-ids:-178648644;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:720442611;
mso-list-template-ids:-911446556;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3
{mso-list-id:1076441205;
mso-list-type:hybrid;
mso-list-template-ids:1926151842 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4
{mso-list-id:2004820809;
mso-list-type:hybrid;
mso-list-template-ids:-1403886720 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l4:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l4:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l4:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l4:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com14tag:blogger.com,1999:blog-6032896665180559.post-34936157443193700332017-07-25T04:58:00.004-07:002017-07-25T04:58:57.723-07:00Nested quota models<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">At the Boston Forum, there were many interesting discussions on models
which could be used for nested quota management (</span><a href="https://etherpad.openstack.org/p/BOS-forum-quotas"><span style="color: blue; font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">https://etherpad.openstack.org/p/BOS-forum-quotas</span></a><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">).<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Some of the background for the use has been explained previously in the
blog (</span><a href="http://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html"><span style="color: blue; font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">http://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html</span></a><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">), but the
subsequent discussions have also led to further review.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">With the agreement to store the quota limits in Keystone<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">(</span><a href="https://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/unified-limits.html"><span style="color: blue; font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">https://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/unified-limits.html</span></a><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">), the
investigations are now focussing on the exact semantics of nested project
quotas. This becomes especially true as the nesting levels go beyond 2.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">There are a variety of different perspectives on the complex problem
such that there is not yet a consensus on the right model. The policy itself
should be replaceable so that different use cases can implement alternative
algorithms according to their needs.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The question we are faced with is the default policy engine to
implement. This is a description of some scenarios considered at CERN.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The following use cases in the context of the CERN cloud<o:p></o:p></span></div>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-list: l3 level1 lfo1; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">An LHC experiment, such as ATLAS, is given a
pledge of resources of X vCPUs. They should decide the priority of
allocation of resources across their working groups (e.g. ATLAS Higgs
studies) and their applications. These allocations between different ATLAS
teams should not be arbitrated by the CERN cloud provider but more within
the envelope of the allocation for the experiment and the ratio of
capacity decided by the ATLAS experiment. This would produce typically
around 50 child projects for each parent and a nesting level of 2 to 3.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l3 level1 lfo1; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">A new user to the CERN cloud is allocated a
limit for small resource (up to 10 cores typically) for testing and
prototyping. This means a new user can experiment with how to create VMs,
use containers through Magnum and Ceph shares and block storage. This can
lead to under-utilised resources (such as tests where the resources have
not been deleted afterwards) or inappropriate usage such as
'crowd-sourcing' resources for production. With thousands of users of the
CERN cloud, this can become a significant share of the cloud. Using nested
projects, these users could be associated with an experiment and a total
cap placed on their usage. The experiment would then arbitrate between
different users. This would give up to 400 child projects per parent and a
nesting level of 2.<o:p></o:p></span></li>
</ol>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">We would not expect nesting levels to exceed 3 based on the current
scenarios.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Currently, we have around 3,400 projects in the CERN cloud.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">A similar use case is for the channel partner models for some SMEs using
the cloud where the SME owns the parent project with the cloud provider and
then allocates out to customers of the SME's services (such as a useful
dataset) and charges the customers an uplift on the cloud providers standard
pricing to cover their development and operations costs.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Looking through the review of the different options (</span><a href="https://review.openstack.org/#/c/441203/"><span style="color: blue; font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">https://review.openstack.org/#/c/441203/</span></a><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">), from
the CERN perspective,<o:p></o:p></span></div>
<ul type="disc">
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">CERN child projects would be initially created
with a limit of 0. Having more than 0 would mean potentially that projects
could not be created in the event of the parent project quota being
exceeded.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">We would like to have resources allocated at
any level of the tree. If I have a project which I want to split into two
sub-projects, I would need to create the two sub-projects and then arrange
to move/re-create the VMs. Requiring the resources to only be in the
leaves would make this operation difficult to ensure application
availability (thus I agree with Chet in the review).<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Overbooking should not be permitted in the
default policy engine. Thus, the limits on the child projects should sum
up to less than the limit on the parent. This was a topic of much debate
in the CERN team but it was felt that permitting overbooking would require
a full traversal of the tree for each new resource creation which would be
very expensive in cases like the Personal tenants. It also makes the
limits on a project visible to the user of that project rather than seeing
an out of quota error because a project higher up in the tree has a
restriction.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">The limits for a project should be set, at
minimum, by the parent project administrator. It is not clear for CERN
that there would be a use case that, in a 3 or more level tree, the
administrators higher up the tree than the parent project would need to be
able to change a lower project limits. A policy.json for setting the limit
would allow a mixture of implementations if needed.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">It should be possible for an administrator to
lower the limit on a child project below the current usage. This allows a
resource co-ordinator to make the decision regarding resource allocation
and inform the child project administrators to proceed with implementation.
Any project with usage over its limit would not be able to create new
resources. This would also be the natural semantics in the unified limits
proposal where the limits moved to Keystone and avoid having callbacks to
the relevant project when changes are made to the limits.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Each project would have one parent in a tree
like structure<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">There is no need for user_id limits so the
only unit to consider is the project. The one use case where we had been
considering on using this is now replaced by the 2nd example given above.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level1 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Given the implementation constraints, these
can be parts of a new API but <o:p></o:p></span></li>
<ul type="circle">
<li class="MsoNormal" style="mso-list: l2 level2 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 72.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Limits would be stored in Keystone and thus
any call back to Nova, Cinder, Swift, … would be discouraged<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l2 level2 lfo2; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 72.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">A chatty protocol on resource creation which
required multiple iterations with the service is non-ideal. <o:p></o:p></span></li>
</ul>
</ul>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Within the options described in the review, this comes near to
the Strict Hierarchy Default Closed model.</span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">For consistency, I'll define the following terms:<o:p></o:p></span></div>
<ul type="disc">
<li class="MsoNormal" style="mso-list: l0 level1 lfo3; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Limit is the maximum amount of a particular
resource which can be consumed by a project. This is assumed to be stored
in Keystone as an attribute of the project.<o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l0 level1 lfo3; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Used is the resources actually used by the
project itself. <o:p></o:p></span></li>
<li class="MsoNormal" style="mso-list: l0 level1 lfo3; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; tab-stops: list 36.0pt;"><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">Hierarchical Usage is the usage of all of the
child projects below the project in the tree.<o:p></o:p></span></li>
</ul>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">To give an example, the ATLAS project has a limit (<b>L</b>) of 100
cores but no resources used (<b style="mso-bidi-font-weight: normal;">U</b>)
inside that project. However, it has several child projects, Physics and Operations
which have resources summing into the hierarchical usage (<b style="mso-bidi-font-weight: normal;">HU</b>). These each have two child projects with resources allocated
and limits.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-QdqdaAlOAsw/WXcyPql6cZI/AAAAAAAAT6A/LKGTXFvyxckZk8hPu-E9Z9HR2A3LqaNcwCLcBGAs/s1600/nqmodel.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="537" data-original-width="940" height="363" src="https://2.bp.blogspot.com/-QdqdaAlOAsw/WXcyPql6cZI/AAAAAAAAT6A/LKGTXFvyxckZk8hPu-E9Z9HR2A3LqaNcwCLcBGAs/s640/nqmodel.png" width="640" /></a></div>
<br />
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB; mso-no-proof: yes;"><!--[if gte vml 1]><v:shapetype id="_x0000_t75"
coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe"
filled="f" stroked="f">
<v:stroke joinstyle="miter"/>
<v:formulas>
<v:f eqn="if lineDrawn pixelLineWidth 0"/>
<v:f eqn="sum @0 1 0"/>
<v:f eqn="sum 0 0 @1"/>
<v:f eqn="prod @2 1 2"/>
<v:f eqn="prod @3 21600 pixelWidth"/>
<v:f eqn="prod @3 21600 pixelHeight"/>
<v:f eqn="sum @0 0 1"/>
<v:f eqn="prod @6 1 2"/>
<v:f eqn="prod @7 21600 pixelWidth"/>
<v:f eqn="sum @8 21600 0"/>
<v:f eqn="prod @7 21600 pixelHeight"/>
<v:f eqn="sum @10 21600 0"/>
</v:formulas>
<v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/>
<o:lock v:ext="edit" aspectratio="t"/>
</v:shapetype><v:shape id="Diagram_x0020_1" o:spid="_x0000_i1025" type="#_x0000_t75"
style='width:451pt;height:258pt;visibility:visible' o:gfxdata="UEsDBBQABgAIAAAAIQBogTHSWQEAANEEAAATAAAAW0NvbnRlbnRfVHlwZXNdLnhtbLSU30rDMBTG
7wXfoeRW2mxeiMjaXWx6JSo4H+CYnHZh+WeSzvXtzdpOVMZowd2UNMn3fT8OJ2c23ymZbNF5YXRO
ptmEJKiZ4UJXOXlbPaS3JPEBNAdpNOakQU/mxeXFbNVY9ElUa5+TdQj2jlLP1qjAZ8aijielcQpC
/HUVtcA2UCG9nkxuKDM6oA5p2HuQYrbEEmoZkvtd3O5Iopwki+7ePionYK0UDEIEpftTelTnUPoT
wq3mf+jSniyLytbcr4X1V33CcyyNExyTF3DhCVTkoEwK+27AccodfMZC+cNimp3GPpJuylIw5IbV
KlYk6x0HxguoHKgYDwH+LVvJjHfGy2g7kkRCY+pwBpbH1ngkzUct2OY1NBLPQNT6jgRiRhrnzwCz
aI1H0vTNNgRH+bTr1EOH/uiS7hEMyw5xQCBtv0Nifw+SHuH7sbQ2h1zaDqTiCwAA//8DAFBLAwQU
AAYACAAAACEArTA/8cEAAAAyAQAACwAAAF9yZWxzLy5yZWxzhI/NCsIwEITvgu8Q9m7TehCRpr2I
4FX0AdZk2wbbJGTj39ubi6AgeJtl2G9m6vYxjeJGka13CqqiBEFOe2Ndr+B03C3WIDihMzh6Rwqe
xNA281l9oBFTfuLBBhaZ4ljBkFLYSMl6oAm58IFcdjofJ0z5jL0MqC/Yk1yW5UrGTwY0X0yxNwri
3lQgjs+Qk/+zfddZTVuvrxO59CNCmoj3vCwjMfaUFOjRhrPHaN4Wv0VV5OYgm1p+LW1eAAAA//8D
AFBLAwQUAAYACAAAACEAtGfdfIwaAABKnQAAHAAAAGNsaXBib2FyZC9kaWFncmFtcy9kYXRhMS54
bWzcnWtvG0eWhr8vsP9B0PeO6tJVXW2MZ1CXLkyAzCaYJDMLLBYLRqJtYSRRSyq3Hcx/36dosU3S
ElOULErODGArtNnuOlXn/p63/vCnXy4vjn6azhfns6vXx/ILcXw0vTqdnZ1fvX19/P13uXHHR4ub
ydXZ5GJ2NX19/Ot0cfynP/77v/3h7O3lq7PJzeQvs7PpxRFPuVq84rPXx+9ubq5fnZwsTt9NLyeL
L2bX0yv+9M1sfjm54T/nb0/O5pOfef7lxYkSwp6cnU/ezieXx7cPmTzgEZeT86vjPy7f6frmq8XN
6sejy/J2X569Pv6nMu2gfC+b3ljVCNXqJrSdaNo+hNSKEJ33/zo+uvn1mlWezU5Xj5t/O705uph9
x+flOT/Or15dnp/OZ4vZm5svTmeXJ7M3b85Pp7e/TeY3ZVHmxJ1cTH6d/XhzwoLjOz6WxzwlTm7K
Q46P/nex/wP/98fz038sbn69mJ4szi+vL6Ztec7tI99/cnx0+oAHn84uZvPFyeT0dHp1I/9Hlafc
Pvb2s+Oj63cXZ3MOyPHJe9kurr+Z3/6ItCevfpid/br8ZPLqYnHz7fIty+fX5Zfp1dk3k/nkr9/M
jy4m5WBNr5rvvy2POln+jZNymHjM8vfru3ZPyN4ap2xjQpcb6dvYuGTbJlqptQi9aKP+18aWLV/4
u9fH//Xd9Jeb/37Qa8/Lu8+3Xvro7Hx+8+vrYxRlcXkTL6YT9EaU509Ygv/uK/9tWVVZDV8un/6w
53P43u1X9v/3v3othTj6fusN7n6BO17/tTj68/evrdn6/j3790EUe+ykc7KVsbeN7F3X9EObmpCV
aERSPrvko8151MPryfy7+eRqwYn85Wqpxk51WQSl+ftDbqxt26YPvm+ij0PQuneidRsH4faQHvi8
9jINMbS56a1yjeR4Nn2KopFdlB1vGbT7sMrF+Q+f5yqDdTmpVjbdEF3Tda1tenan8VmlKL0xyoeN
zXhvRg6uld+8+3VxfrrYOtWH1csWtXxtinqpp1Qvq4zOPqmmQ5Wa7JdHMORG2870Muc2SfTj1s19
rF55kF3nU9dow5E1gr31gge1yfjBxCh8SBs7+jzqZVNwg3Rt04XsmyRa0Tith8YpNWBeTDKG17xd
5cfq9ZmsMjvnBf6tab2Wjc5t37h2iI2W3aCz9SnLuLEZz6ReX19P55MbAsjn1TBbNGzpwPRTalgK
OogkDYphciNaNIyzpxsTUyt7b1o9fAgkP9Ywk1RyfcqNb6VqlOboBj1gQlsxtMK1zqtNm/k8GhZy
1zlcWGOFtE3o26HpVS8aL7OxprdJFz97r4Z9JqvsvRB58BITElhbJJoItjWNEAOOrW9tNmJDwx6y
GXdHXzvDyD+fv327qUzvA+XbePoBT/zqtSrqIZceSGx5wk8Z3xnjeiEI0vuhD8WN8FPrcpM7YYxp
bT+Ebjw4H6tHbGMfg8QBKdU2NpVzl4j7BzuY4INXXUS7bvO7kps9ZEfuWS+Pei/m385HUIE+Y4Z5
wxAb74oWh1Q8ZglBOzXktSj2Ywf0mawyd0G3vZSNKrF6y/8abwM/JZIvYtrOpcfbqgcc5m/PL3+8
WHqcrYN82JBOLjVq6XGeUqNcF1vtRd9k7X0jY+sbNIGYrE2tdbJXhGY7NEp3wZkcSZ2zTo3t2EQ3
GNEMMg4ueKdM374AjWqlMFLmrhnarmtUS/gZlDIsNRvXqaSwAOMqP9aoz2SV1mEC3aAaIrrQyIGQ
LlhLMSMm71OKQ8rPolF/n83/8eZi9vMz6tPRV691USh+IUl6SoXqB2UlJaOm8+QM0bAJPrhEKdCa
IffJuNCPR+1jF8X5LCUH3WRJWI5y4gCsNY01WqR+SDr05gUoVFQ2eNK9RtrosR2YjX4w6Fc3WN/2
HLadEdznsspkiblj32hhVKPbtmMvhwEzqYTFZFitH2/dHuCi/j794Rl16VaVnj7Y0xjnQDKjNEE0
to1iXk8y01rps03S+lKWvbfaMDgtpWK38EOpGVTrml46XeplqeuJsGy7WdV9nmBv6Izqo2RtXZ+J
84prSoLKSO5NK9Cz3O9yTZ/JKlMWyUQijJgo5slUVjlQvCzxbDRKEmmsWcU5faC1KPzomg/8YjE7
/TLVN1vKl/5jcknD5d35dB7fnV+c0SYpny4bCPHqppTWV//MdPHVsqXyt8l88f6zVW/l6KfJxVpv
4vTdN/Ppm60PybduT8/k6vzy66vp+z//gfLy6btVe6D80Vc/0RArz7v46WL1OW90/n/TP9MGu5gu
SppQGhblPdff6PZDcpL3b3fgkjMRU9vaQTR26Amb+JnC+kCwHr0OMRpOq/qgi7z7KNjS4SqL+bB/
de2W5Zc+7N9fZ7PS5arfvrLpYSn/9wI/vzq/KRLfR7rv/+6dLSPqFLTiukYGMzSdJRHFM1CxGajT
UB+0Nq8lop9YHnNkEWeX17PF+c30DqFsGrL7F+GMptzUUckVJhHBtwojhJXF2QkVqM9GX62Ue29q
WUTpnq2//1c/oBlXdFXpf407/eXZL8uG2PjBUnP52umP9AJPJxfT/0Q9yYydGw/dpu4UfVpXW73v
OSgP4BQ8eSOS1hPbUXyUoRJLyY+KkSd2jFIKZcSgxSCfTMven6qrq+npzWx+566sf3j3rixblUVa
BzZPwQ8uaeWJEiIFhdIRC4F2/KBVcE5b1fUfSvblII0n5ROZp6V7obe9eUi3xXG/KlLYyn3Pfueh
JwNvS7BC9NMEg9eUOqgQbO3OV3Zpyqu+t6/lV92tvfxSD8lN4rtl61Cm9ZXdvfNq5c0OvPNayyxb
PFH0yZdaGdmvCwAxnAwh0uEQljbGKkjcvfN1Hcc7HNO6eH4zrnhaxxS8N7pAT8hTQjN4EjMniX9T
TJ3DuitbCjJPJI8Nx7R2nm5lUuuXtO8HSz+wSRq/lAXBhksdXfmByL8Vyg/iQ0j8G9q8956u/NLa
649uaX2f71OD36Fbkom+RQsgJvs+NsNQjFMpm0sFzELFdhjS0+5HnN26pYduyubJO5Q7pyUeo7OJ
HgUVnDa3oQlSRQreVBxx9Er0T2ubll4JINdDvZLssaDaUYfqM74olVpWTrTtKBG3RCadsdVRbmXr
Zh+vpNdWdrc6ts/klWSnnLS0qCS4oabLaEyIKJCMxljagS6ltTbubq9U10Z82V6J5N8FI2nUiYQ6
FCyPpxHXmA6wAX7JgfOp9Up7y+MTeaU0JJO64k/BujTZEWO6lqY1mVLf+b7Hs+YnXUPJltZO/OiV
atRg0ytJ0ZluDIE/02RpkBghlUyTiP6bDgRuA/zP0fwIJnY6ZtBiT7ofO71SzaY8j1fC92TXDUPD
gXaN1SVXAviKQuY2e7AJTqyVVZ/ANj3WK7lonYtmoO1AASaF0h4f8kCvKxEfkjUT9D7Zxo+1RLOm
invGt4B4aI+YArjCnFBnKY1ITcLPFgilewox1cleZf/2UW51Pe9fLvW53OqQeu+zVk3MmRY1aTKC
60LjSP8jkYlSqjoeqYMfvGy3mujOU36lqh41UpAFIhtj1+SsbBw6LUSqdkl7y+MTuVV6naJVAiSO
GPpShKThyds3ySg6jD3AUEHDsy5hfdAaqt3qHWrwO3SrFHJ70YOR6kEZNa0gaSEbx1G0mjpaC2C3
vij8oP3Yz63esSnP41Yz4X7ywJkAmmCbyjnurfBN5zofiXp1Z6tD3L3lNnqlRyR7lPxxSux0ELRb
SesL9Iy2d/Zdlrb1IbvqbsDDF/AIt0oFWFumPJrYZduorsDLtWUIyxILimjkkKoDwr3LRuMOPGIB
BuRu8BSwfQrMOegyRUb1lwxDAOol2QBwUGsKK4HE+8QFH1W/PtK9ZysCS69d1D3RLKhJoUkFgqOl
nkE+0eOinSWry/91uPiXHRd0HPvWSA1yIpIf+VIUJz2leBOlIvGmu1TvU6vmBNbk8YniAkN+x5EH
wiatBsJWSiiBWRNvMnsqVEd9pVYZ9t7TvYrAd6jBZlzwu+hNLqFQXU5MYsm+kS1nqreZqJPRGKYk
rbD+A2a/mJWxvvBxi+1B+7EzLni5tolG1BBLK8/QqWqGXLxS8tS/RN/2LlOlcE8nt9ErPSIuoDHp
wSh1jZVm6ZUMG29oUrpgB6+YbPbV0I9KcOQ+XumjQsv6UXjWbFW02YEZZMKqp5vPMGqHGQMlHaQG
HukB+cfq8n8dtHfNCped/ysQi4+ksQvyVL70dJgZN+jU0pZtRJnQ7YYSYA6oQ1Yg+qTItOqrs/e9
5fGJvJJKIA+NK90cZnraZcbdAiSn+ODaQGcy66fb011eqUYNfodeCeBSto7QRmlwTL3lOIXA6CZd
K9srFYBPVhunB56pHa3Jmk15nmw1WhGplhc3lNqSKyVgshxrkJjZDd6KbqiuHO0tt0/hlZQMQ6vB
OYiYMCEdIb9X4BMBoQd+DzL46nT74Qt4RLLXM445SKwh6QlAJYGfcJr2utRO5DB4ukoMa9TVvZjK
rkFKP8qtrp/lZ3WrLaqey0QFCg4GQZfquRUEJdGT5/UmxFAtuFgF8H/ZblXnHsAP/XndkvK2pqA4
CxROtcyL9A6bmKqTvb3l8cncKnMjRhsK2QW6RN2XNYSeqWApdEIhcLe1yvCgNVQXge9Qg9+hW80w
p6hCLmIdJcwlnxGjfKnMByZQP7JN9ai6B+3HzmRvfQ+WuI/1D57VNoH4gSGC2WLyOlNIFTRmKdmG
SeM+GuejWR9Z2Z0k7y23T+FWo+07oEnkqAzhw+tRqtiOiY020jSOLg9JVlddHr6AR7hVzB9lbFGK
ReCoYcmBvMPT5zai1wQ3HdM/1dZw7zLFuAOPWEDP4KxoGQVKMI9ARETdiwy8ZQc6YlkMou2rq9h7
g/LHBXykUfdCmW8xzSMvGlxK6z+vkaSxAEmzmx6lJR+2EFIwkEWRksaJdWB2e5E2KyFfv4EGa366
pGaqBDWfTRdLErR/VgKqefzX87PlfEH56urnEimVf/YRhGy3cT3iWBOBDJphTtYsSxEtuoImImpp
gAYyNlJQGXlTvdZFUAl8GkVQCVg6sAjqZqnHfa9sxIyLruw8rRZN7XZj328pwd6zgdWNxq/GxZff
qRs0v/Ns1NE2PFgwladnJRgmcO4VTCWUc0MwdZwWdwqGPEnkONgyO8jwPqQReCdiE6WcIOGHuW+r
ALmuNJVOaDw/KlVFoveJ6YnshjHZeKpLQODBCXVAnKgngqQkAQ62Tzjsrebyuggqi66jCCoLvgcW
gSSzaR2tICpsqgzxFr8oIL0LtB9MEtq7Tb+4LoLKBHkUQWVyfmARQEfWkfQa1myZPMwUlPtQhCG1
CQrGDTNsFlDXRVCp/6MIKuGVKxFsG9InUoQ64oDRSFZGcOOiKytBq0XvMpKVWrdhJOtYFe40kgoG
Plgn4JkCo9PEpXp4UARKiwIb8OAi7z8blfS1o5gqp7nvE9NTnY0MBS+tqob4v286iYXwmEqmiCiR
UDSiV7VZC15Xj0qfNoqgcjTiwCLgrXrJDDLZAhxy1MYKrUDCbYAP7Af6LlLdH1/uqyyVHeGVCA5k
IeoIER5sISrjifsWvR5fVjqlDQtRxxZxp4Wo46UcBVOZO44KURmorwSzy3RW5nobgqkj7bxTMD3w
KglvEkkZdiMQZgC7KmXICDNU10bTbfWQ1u1G5bJHMVXONt4npicynW7oGZUGeQbdClxD0C9iN8pQ
rRigWqGxkrYm1dZFsK/dqET2HFgEsG4Cx2BYz+RCxhJKiK3pMjBXEgqynImSzcHZdRHsrSx1g/4r
ERzIdCb68gJkFXwD9FENtV5K8UgEDGLqOgQ0bEFV1kVQCTgcFaES7LgSwba9eCJFoJcMhLCgzkJf
eBdQBJiCcgPbdCcdnNtGbSIJ10VQ6RtGEVRW2VciONAp0FxhIArltmO8txmYsQKaT+JtyTXhKEIG
w/1lun0VoZKCZCWCA52CrssmyIIkzpDB90vqqAEotAVjauit8af3i6CyAjWeAkRWA7hfieBAp6DV
AX3HFXp4NQrVN/31LD0DAQZQaQcD+w5zWJlGjSKoRNWsRHCgU1B3G8AYMO2bRFWqyn2LXo8kK689
2AiY6i4RuDNgiowIlmo1LQqmmwuLK6SfgBiMZn6g5ZjQed7o1q4byX3PRiXCZyWmA6mHjtJamA4L
qpNLIkqjj+tf2gZSawXpiLUULe8Vwb4xYyX/woFFkHSGJB/MEHAVyrKFHKA35JrA4Mg5fYa46H4R
VI4tjhaicmRyJYIDWYiUIcOgMAvEntoslWlQVAb2W+3L/6HvozZ57ynY109UDsodWAR1XOyjkaw0
euO+V+YWq0Vva/+6kawMUDeMZB1R/Z1GsoOQDPYhoF1MJJeSPbwHrmNEBd6DUjwCsbZpJL+ZzNft
ZGWNbZRUJUPfSlKH0pBAHGUVoZODwAdJULKG95sRrQg5vpKDkZs1uS0pVK5qlIKo4+U7sBQMzL8c
JdmUAhzYkHIWVCayJLUAqBjArm7mlltSqFzVKIWXGVWXIzDYAJlT4UykWA1QrfdAMyIpR8e1TbLf
LFE/UgqVbHqrs7BtO54qw7QR9WeQqeQYpVpPiC0g6BqMZTaEcGIYNhHkW1LYVyMgx6ihxjuwFHJP
MYHhPIjiE+BtX+IniQ+lCBUCbMtt3JrV3JJC5apGjXiZcJDo4MsmaIYgCuwjeB8KcC0YYF/mqQau
gHFrtPQFk/JIKVSS5R34LBh4LaUDu01MTeGlK5d66DIhklxWKiq5TZa0JYXKVY1noZIibyWFA3lK
bwim4UKmcUW9ieARKTAe0+TImBiz96bb4trakkLlqkYpvMxCdPAM63qYoYShkQczBCVIwO2YSDDN
aAXUPptJxSOl8DJTq8EwJzWUGyglN5C15SKHfsgCTgx6FSZkzsJmXrElhX01opKobqURB/KUqu0x
DbDvcxsjngEcO5d2ePJNpw3DE5Aulwti1iYktqRQuapRI15mS3fQPkO/COZJ+w4BlClCqNQJoriT
suu5DbFc6fUJpVBHXXfgswBjHGogyoxMohQdkQLXhknSbR2kstzdJHbGjpWEfONZqCSsW0nhQD6i
EPqEPnCtiC7sUX3xERTrm5yoyCnHlWpb5K5bGlG5qlEKLxNBarlODbKg3OAZVbnYodTeINgsg9aw
AwRJCLVLI/aVwstEQjn4TbTVZFMWLqzl9RYOxjuat1bC8pC4CXBzgm/rLOyrEZW0cSuNOJCP4Fqj
MoS75DcmgBSlvtAy/0nnjsHQ0sQbdtYX9pVCJffcSgpMXxdlempYeYBRk6s8y3Q+zFnQjZJZC0b0
XeLGkw7WD25Z3aUR+3rKPauxB5ICCSR075wFzyRgowN2wQ2OiyoMdWkPZBKV+ZRSqGSjW50FRioO
cRYcd/HCh45JgDil5JQ07hxXbUZQNSSU0uTd1rFyVR98RB0H3UoKB/KUcfCu55ogJitg4SV+Imoq
NyqGCJ2xpKUJHcOus1DJrDdK4WUW51vPwA1GgCiBghMXHlFrYrqbo2EyNysBcjE784h9pfAyW9mA
GUgkYLTzAv8I1TDpNRUHSEON7KUF2pJ35pT7akQlE9xKIw7kKYPR9GgdAUJvGQ7m8raSQsRy2Utv
bZHBbo3YVwqVdHIrKRzIRwST6L9opJAlswPlHqM+RZg8IBQtMxYyiZ1dmX0z60pOugNLoWRTTM9w
j7vgl6gYoecWT9MYkC5S4DTlFj35Vuy4d/W1jtjuwFIAEe6c4qpFckdyqICnDJ2i3ACGwTJb0kLn
uMtH7CsFXcdSt5LCgeIFkkgfFdSrvYVJlxNQsG/8xD004B4BQ7Zb6Nets1C5qtFTVnLTraRwoHgh
dwOWIXEMCq8P2KaWUguc0rEFBtb3wID1zipL5apGKbxMDGw3pE5Au1eu8cQu9AX6Ry0SVhjiCOgo
XWt3ZlP7SuFlThAYQI42Mlc1GNwliCYI0hV4SDUwQsLVNMxkb2JAH6kRlQxxK404ULygs6YSmCCw
9QSQ4BewCwC/ypUN3HYXKMm63TllHe/dqBEvc+qQsKiTTFbRkOFqgY7bh8r1UoROCaZ8sm785c66
Y+XejlKo5Iw78Fko4TN3L7km4Q4a4QtzW6nDdtESS9HC/a3OfR0T3iiFSqa4lRQO5CO4QEv63tCo
teB4HJweaATYHlsoVak8Gd9tooK37ELlqkYpvExULAxLuMqBYJGbnhsgkEihoMVjoODQ62ThrdgV
Ne0rhZeJ/+TmI5iVuAcmcNFnAT+SWIaOq4o7RnFtQRt0O61jpZ6PZ6GSr22lEQfyETYAAoWNq0kO
znPuhyJeiNSauHGxG+hGJK4s33UW9pVCJenbSgoHyinJH8sdaDjJ1kJSUJiKcZkYSy6r5pRIpXZ3
7vf1EXsOpx9ICsrSlfPcWa+z800qzJje29xobmvppHIWl7nrLOwrhUoauNVZOFA21UKu3SWk0HbQ
PoHoohKfmKhS1KCysR76bbtLCpWrGu1CJfnbSgoH8pRQuVFRim2jgTmiB6Urwx22NPKBg2qchHG7
I+g6SrtRCi+TtCMBUOjo0XPpGUSF3VIK4OIJH4ij9UBXJu70lJV7O0rhZY6Ttdx4pxR8VqUbV7oy
dO5hp28gpecgUIvPW1fzbkVN+2pEJQXbSiMO5CmJG4URAj4GbqdqoP6mN0XNgelSgD29jMR6O2tN
+0qhksdtJYUD+YgeCjO2nFoTdC0Nd7kyZqy4qwNUkxwKdwNHeJd13LfWVEkGd2ApqERQUC73yECB
y2gdFUiS6oJl6XRWSfgtkoZtjWgpT1nwknaAZJBLsGhxkqbC+VAug4rGw3aBFEe7UMko96RSWLLE
rVPD/fD2dmri53ezi+ntz1xwuOSOm7zip6Mf5+evj9/d3Fy/OjlZnL6bXk4WX1yen85ni9mbmy9O
Z5cnszdvzk+nJ2fzyc/nV29PlBDu5Ox88nY+uSy3DyyuX51NbiZ/4UL5i4EH/nJ5cbV4xcePeOzR
fHpRmL7mX55B8nd5fvW36fyjx82up1f8Y29m88vJzeKL2fzt6h0vL8pb2vEtWfjJcrX8fsa96SsJ
LP9jfPk//j8AAAD//wMAUEsDBBQABgAIAAAAIQBGAF7ajwEAAEgEAAAfAAAAY2xpcGJvYXJkL2Ry
YXdpbmdzL2RyYXdpbmcxLnhtbKRTy07DMBC8I/EPlu80j9KWRk17oBT1ApWAD1jZTmIRO8FOQ/v3
rNNUCQUJ1F6i9do7OzuzmS12Kie1MFYWOqbBwKdEaFZwqdOYvr2ubu4osRVoDnmhRUz3wtLF/Ppq
BlFqoMwkI4igbQQxzaqqjDzPskwosIOiFBrvksIoqPBoUo8b+ERklXuh7489BVLTeQe1hArI1sgz
oPKCvQt+D7oGi5A5i/qZlmPOLkeGSNePpnwpN8YxZ0/1xhDJY4rKaVAoEfXai/YZHr2TqrQD2CVG
ufdFkpBdg7J33wZD7CrCMDmahJOJjw0Y3g3DcTCdjtou2fMvdSx7+KMSCR0aY9Aj01q6MjiII+VG
bUxuMqcTB8eJlxKwUpGgN3q/rFOgn3UN2oa98IId4AcaaD9PVWREvua2dR4TZ1h/BGxBzH8g0EfJ
xLJgWyV0ddhzpAIV/mA2k6WlxEQc2Zg1RwEN7mkThy7+sE08dDE7xLdO08alRjsnT//slLtkhb4v
QmcP9jj5h/pdf7D4AgAA//8DAFBLAwQUAAYACAAAACEAaOY32DIBAAD8AwAAKgAAAGNsaXBib2Fy
ZC9kcmF3aW5ncy9fcmVscy9kcmF3aW5nMS54bWwucmVsc7STUU/DIBSF3038D4R3oZ06F7N2DzYm
S3xR5w+4obQlK9wKTO2/Fzu32KTtXtwLCZxw+HLuYbn60jX5kNYpNAmNWUSJNAJzZcqEvm0erxaU
OA8mhxqNTGgrHV2llxfLF1mDD5dcpRpHgotxCa28b+45d6KSGhzDRpqgFGg1+LC1JW9AbKGUfBZF
c27/etC050nWeULtOr+mZNM24eXT3lgUSsgMxU5L4wee4LmC0oJ+3imxffVtLYM52FL6hDJ2UB1/
P+oxC/yUD6PdnAHtAWu0bhhLdNok0u0IklbCosPCM4Ga74P6CeiuP4NDBJmFz9CAYYx8L05yzEc4
Bhpxemo+tKk/qe6Ed+skRvyfGL/lycDDSDBBmcSZnQHnCVrc+WGgutOOSLz3Z9NvAAAA//8DAFBL
AwQUAAYACAAAACEAuiNWF0EGAAANGgAAGgAAAGNsaXBib2FyZC90aGVtZS90aGVtZTEueG1s7FlL
j9s2EL4X6H8QdG/Wb8eLeINdP3bbrJMgdtLmSEu0xJgSBZLejW/9BQUKpEUvBXrroZcA7W9K0aY/
okPqYdKmsw9sgaDIGlhIo2+Gw5nRNyT14OHrhHoXmAvC0r5fv1fzPZwGLCRp1Pefz8Zf3Pc9IVEa
IspS3PfXWPgPjz7/7AE6DCjJ5gzxcBbjBHtgKBWHqO/HUmaHBwciADES91iGU3i2YDxBEm55dBBy
dAkDJPSgUat1DhJEUv8ILEplaEThXyqFEgSUT5UZ7KUogdGfLBYkwBobLusKIdZiQLl3gWjfB5sh
u5zh19L3KBISHvT9mv7zD44eHKDDQonKPbqG3lj/FXqFQrhs6DF5NK8GbbXarc5xZV8DqNzFjbqj
zqhT2dMAFAQw09wX22a3MWgVWAOUXzpsD7vDZt3CG/abOz4ft9XPwmtQbr+1gx+PBxBFC69BOb69
g2+f9E6Gtn0NyvGdHXy3djxsdS37GhRTki530LV2pzkoZ1tBFoyeOeG9dmvcbRTGNyiohqq61BAL
lsp9tZagV4yPAaCAFEmSenKd4QUKoCYHiJI5J945iWIovAylTIC41qiNa034r34tfaUzig4xMrSV
X+CJ2BEpfzwRcJLJvv8VWPUNyMuVd8pkTIJiVG3E0jhDaWRqvP/1+39+/tb7+/df3r/5IR90Gy9M
/BCn0TcEpR8aAGa7CcO7H9/++cfbdz9999dvbxz2jzmam/AZSbDwHuNL7xlLYHKOGeA5v5nGLEbE
1DhOI4FSpEZx2B9B/Ez04zWiyIE7gUiYuBccaMYFPF29shyexnwlicPiozixgBPG6Anjzig8UmMZ
YZ6t0sg9OF+ZuGcIXbjGHqDUyvNolQG/EpfJQYwtN59SlEoU4RRLTz1jS4wds3tJiBXXCQk4E2wh
vZfEO0HEGZIZmVvVtFE6IwnkZe1yEPJtxWbywjth1DXrIb6wkfB2IOpwfoapFcZTtJIocZmcoYSa
AT9HMnY5OV3zwMSNhIRMR5gybxRiIVw6TzjM10j6I6AYd9ondJ3YSC7J0mXzHDFmIodsOYhRkrmw
U5LGJvZLsYQSRd5TJl3wCbPfEHUPeQDy2JfuFwRb6b6aDZ4Du5oubQpEPVlxRy5PMbPqd7qmC4Q1
1QD5W5yekPRKgt+i9vZ/R+0TkgYxc8zorkjdbdrKyA3p/JgT5/t0tkXi+3Db1D1gPCQfP3MP0Sp9
iuFl2W1fn4j7E3H7/3vi3vc+3z1dbxgayFstXfPFul66J3tX7gtC6VSuKT4XevEuoC+FYxAqPb1D
xdVOLovhUr3JMICFizjSOh5n8msi42mMMljh131lJBKF6Uh4GROw8Ndip22Fp6tkwsJ8w1qvq81p
Th4CyY281q7ksNmQObrT3WzCKvPa20hvlksHlO5NnDAGs51oOpzolkIVJL01h6A5nNAzuxMveg4v
7ivzZap2vADXqqzAwsmD5Vbfb7dABZRgT4UoDlWe8lSX2dXJvMtM7wumVQGwiigrYJPpnvJ17/TU
7PJSu0amLSeMcrOd0JHRPUzEKMRFdSrpddy4aa57m5Ra7qlQ6PGgtDZudO9/yIvb5hr0trmBpiZT
0NS77PudZhtKJkBZ31/Axh8ukwxqR6gFL6IRHJkFkucv/G2YJeNCDpGI84Br0snZICESc4+SpO+r
6VdpoKnmEO1bvQGE8NE61wNa+dicg6TbScaLBQ6kmXZDoiKd3wLD51zhfKrVbw9WmmwF6Z7G4aU3
pyv+DEGJtbt1FcCQCDj/qefRDAkcaFZEtqm/rcZU0K55oqhrKJcjmsWo6CgmmedwTeWVO/quioFx
V8wZAmqEpGiE80g1WDOoVjetukbuw96ue7WSipxBmpueabGK6ppuFrNGKNvAVixv1+QNr8oQA6eZ
HT6n7m3K7ZVct7VOqLoEBLyKn6PrXqMhGK5tBrNcUx7v0rDi7EJq945ygle4dp0mYbB+pzS7Fbeq
RziHA+GtOj/obVctiBblulJH2vVxYoIybx7V+z58IIDTiddwBZ8YfJA1lKyhZHAF3w2gXeSH/X2/
uCgl8DyXVJhmKWmWmFYpaZWSdilpl5JOKen4nj4Vhy8x6kDc98pDb+hhxSF5sbawv+Ac/QsAAP//
AwBQSwMEFAAGAAgAAAAhAAPAojwRBAAA30EAAB4AAABjbGlwYm9hcmQvZGlhZ3JhbXMvY29sb3Jz
MS54bWzsnF1P2zAUhu8n7T9Evh9pGSBWERAfq4SE0KSx68lNnDTCcTLbhfLvZzuf7aClsbs0xdyU
Jsqx8+T49fHxSc8u5gl2nhBlcUo8MDwYAAcRPw1iEnng18P4yylwGIckgDglyAMviIGL88+fzoIo
GfkpTim7QaEjrBA2Esc8MOU8G7ku86cogewgzRARZ8OUJpCLrzRyAwqfhf0Eu4eDwYkbxDCiMAGF
EdjCRAJjApwZif/M0G3ggRkloyT2acrSkB/4aeKmYRj7qPiAlMumj91TN78FF/o+Inz4+xCcqzvj
McfIeYLYA8DNDwWI+YtHfMjvGM/Piv8d/pIJQoUp4GQ0FkSHoiFpwlXAGlcw/oLR3QQ7BCbiMpIG
aFC0HsYYX2MqjDsJ4lMPUJQhyMVZOFJckTib96VsrWyhvjTvF47JOy1hPqz6WV2VG0FhiHye2ylo
8Pldabk6Mq66XR36vnilglDeeG67/FZggDiOyL1gMeySxTLV7nhIt+gUxW64Be7eJ3YDxBMiuiik
ikCcTWGuIMcD8adGfkNcCr3qu5rI0VNKen0vH05UJYavXerpbowdieHIYgij2yT6gaGvM7FIEeEx
4X3RkLVjvumifF6HMnmUwuftIhlL+t+Q1TjpiXXoV1YGxjGzePJAIWGHNxoB6aJunGwl9iij9/e3
1V18H0Y/LVa11K3WjLX2Voc2Ft+JxVqs+yuGJrCWEjDUkQDRI5UHqAPyTVMV5QAvLekO37XBAZ/X
uREjwYEPMU5nMqMj00I1ibU9Wb71+tK2EBdVctVqsH+YIWPcJtWAxKAxZ5fDrL2vNWMRXS+qBG3j
SUFisNkA5Q02G6Aw2GwAyCC9ntq4Xuy1VMIyrvYx1k7HTWEzEhg0H0enWrUcaOjKdrcoO9W7/ULZ
qWbuE8odWUSp1NEUBsV296qUzNY1wPg6q5bT4c1OyOmHob0TirtE+3RFurHfvr0TorzXtMPo0vdb
LKJlhCjBNHbcv61ww/Zr7P8+NQaPppNiKRlbyksFXsYpqwor68uyPLEuozNOmdNLWclmjPOR1Yw8
wb1ULzixgrFtwWApjgMdYdbPH/d/blMQNSXBcvSA4nilMegtRA+oWHacii3I53u9+uK3Ittmcdh2
py7dHmx93Wc8tCgDOPsAVRlB/x6gilns0+vp01Pi2aKowCYC5AtN5fRbhvGvBvUKcYt0rUW8GeIW
OVqLeDPELRKzFvG7EYsyz2nWvohOxo71SwzbjVNL4etfuBI8XuljZvUO46pdmB6lv+UwLR+qkdIP
Tg1wrt25quRc2HvYEyc3Dl/U4ZtTklV76O09vHnP/VMRip4e5ptL9SuTofmXVeUSfSmP8FYjWwev
VwKh1lT5bxqIn2U4/wsAAP//AwBQSwMEFAAGAAgAAAAhAGublhQYBAAADVEAACIAAABjbGlwYm9h
cmQvZGlhZ3JhbXMvcXVpY2tTdHlsZTEueG1s7JzbTtswGIDvJ+0dIt+PlMMmVBEQByEhIYQYewDX
cRILxw62C+XtZztp2o1Wc6FRncw3JUn7O7E//0fsnJzNShq9YCEJZwnY3xuBCDPEU8LyBPx6vP52
DCKpIEsh5Qwn4A1LcHb69ctJmpdjqd4ovsJZpBthcqwvJaBQqhrHsUQFLqHc4xVm+tuMixIqfSry
OBXwVTdf0vhgNPoRpwTmApagaQR+oIkSEgaiKSPPU3yTJmAq2LgkSHDJM7WHeBnzLCMIN3+gUObW
3+Pj+HlK0JPtRixJWVF8BE5t1xRRFEcvkCYAxPWlFEv05xUE1a1U9bf6OFJvlR6iuiEQVYLoER0d
jUamhdgM2LKARJjhw/T0BOrLJRZQC0iVAC5UwfWIVAVB14IzZaThmJK8UA8kjwTRZFQhML5XIEqJ
0GftDdpGWzy3Exox3X4CGE/xqOle+7vubl4dps3Aqdm9aA7tUNvusAc9b0g6S4B5JjiWSOSTSyoi
3R89CXUf9efEfJrB0/03AuaHGaG0lT10km1EjDTOMoxUK3/gJN8K2ftrJK18SRgXTRt6wmPTATtp
qNpvHjyrf9/MgGYA7Gywx5pPPYHmZw0tyu40r33dtkXZ/WRx5mWeKfCqjd9Cu14w85bYLjXM7d4f
0DA1+6SGQW1SvUUWlGwRYSyUzLgwH02i2yT/31yYoWUcrG8OLNBap1smnAq0FuGdv+Gh0a15rhKC
+TqR8JdWlt+U+T2FaHDeyy1O6ToBMylim/EuH6/LrmzsF5g0KXUbfW9TjzZmMglKYgobnVUpNgYi
yeRRQCYPrkLM3ZeyUZb/DNQ6VaO6+LXdYt8kUKurup0Zvy6oze3jvpf20S0yW12TcMuQ10d1bvJt
2GHgNGXyOgKpy+obOywEKeVT5WES65aarIbhBnI9DDf5rcOAUiozD0I9oQ/1BEMrRHl9ifIMrVBZ
7ROtUFntE61QWe0LrQqKyyJUKTosG3WROS1TC36sj7rmoz9zS3RWJ1nDXUi2rGs++rVAbdV/4xfU
QnXp70WbO6ouLSPx0Wl9RpHchnR9jclNfus1pmUkQ/NIbkPqNZKhuZueIsnyc4R8LO59xmK5yXqn
Hoiz64Dj/UYMN9XaugOx64CCdrzbF7MjHEqcm0X5AYgvQCbBVnljqySnJA3OY8USkR1ZKwskGKzV
axZ3yeQiWC1vrJZNPq65Xpjzeufp7i23VGJ13dhN1rs0ZB73BjB2M7k/PsUGXIGKZ1SsETMezbfF
bG7mZ1Cmy7IYWv3djaN3bsSyGFrhvc8shlZx7ykLvZ+jqDx0F25JWTfuwpjsf782ZusV3vTpIsB4
V07cEQwlAgyPMvPhGSk/lnFtvHFJ4JfH2cD8hZuv2W1Iu3hpgH6L4elvAAAA//8DAFBLAwQUAAYA
CAAAACEArZJtZmkQAAD3lgAAHgAAAGNsaXBib2FyZC9kaWFncmFtcy9sYXlvdXQxLnhtbOxdWXPb
OBJ+36r9Dyq9TyzKsmWnxpmatTe7W+VJUkm29pmmKItbFKkh6cTJr5/G0TiIhkRQ8inlwdFBgECf
Xx+gfv3tfpkPvqVVnZXFxTB6MxoO0iIpZ1lxezH879f3v5wNB3UTF7M4L4v0YvgjrYe/vfv7336d
3S7f5vGP8q65SucDmKWo38JnF8NF06zeHh3VySJdxvWbcpUW8O28rJZxA2+r26NZFX+H+Zf50Xg0
Oj2aZfFtFS+HcpK4xxTLOCuGg7si+/Mu/c/sYnhXFW+XWVKVdTlv3iTl8qicz7Mklf/FVcNufXJ0
diS2cATLulzAx9HwHd9ZkzV5OvgW5xfD4ZH4aJbWif1JEjfXdSO+hdeD5scKKLTI0iquksWP4WBV
ZUDT0WiEc+irkrIAqjfymlN5zRGjqjltHS9XV3ETyyXAqz/KWZqLtyt9+1UzWLIv2OaBhWIlszLB
G5vf4x5X1Ze0GawW+ayCVbIr+e1XckfmkDFOGdd1I0kUMPw4fMgkfMjJ5iFyg5pr94X5WtPwBKS+
SiQ1gfMNexnxDz9WgsbsU/laSkhyX+gZTtUMMAxnADrCtB1nmFIzHOsZ5Lxr1sB0V+zCWMNEzwDL
2bCLc2oGQRx+X1iOPYOQYIOsN7eSOt8XZZ6ikM20JPMRtpzXzY883b3Yy4WYgh2NSQ05NrTBFBDz
tWa14Oo2whKR0gKLU+KymdkRKS+RITAOu/syy2ZPklePwyuggTBs3AoFcpNZVovzk8fmsbYphjZG
8Lq7SXgiOeFeyTRuEWlZonWmpaesWcIlnPUHcHSDIl5KX3u5yPIZ+rRvcaV0FH26cNrcwfGNLD5V
gFi4b1cfzrJKylNcZMuPhfT9N1VcJAs0Eeyr628AlxgsyL/l+HmV1tnP9N8AkvK0RgtnriRZlGWN
a/4AKx9JV5XN5UbYhyAJ87siuRjC0OEgrgCAwbKGg3J1MUz/vBuK+xYApOToOL81MAenA7rAGACV
/C7Piis2DV/1vCqX10rwYQLp9LO5gBRpbq1zvN2dPhN3YneQNxVkETeuF/EqlQgQ4MhmECkA3VWZ
3C3TgsO506MqzeMGgGy9yFb1cFC9vckzIB5uYvZ/kA3kD7+hxFHAwI9zyX8AZnWjhUi8laT8Dhwq
YXXg8vgrxrWLYVWWzWW5XJV11jAmgnR95UiQXR4nDYOAKCrWdIvg6WDq92oB4g29BnHf0ZsT8sYd
9rHbbdizPd4uwPntkhut6Xa0D4gSlu/Lovny05SHVSOEqABrp22AJaRSKuuVOQ6txeYrQZSF8LCI
5TMIcUt2g7c3VubUEnN7fc5dAZUoHrFLvbfVy+yxN5N129wlu/ki1qhU3Lvetl1AnfSRSUxNGBi2
cW7dLVLxy0lKbJ7IosY2E4HH18zbZiIAR7uZCJD0biYCON1lojT5IuitLgbrCq/D7LRPJtTsncUC
R5CSgV9SbkxJmSUcOKL/dJaIbD+dJSjbT2eJy/bTWUJjTCfQrw0tqrs8FYCEwVJg8D/jZGHgQeBD
fJ/VF0NAoAKouNcAccU1dZrPIZlkOg0xhsTMwtqLC0ygymTgHxzzCriYFVmjMJx5oYNoMfliIVog
bhvR6jtopyYBtdykM/VUfmFNDQGIIo7aNo8L5S2TohF3uG3Af0q0L3Zs42ZGC3mHlQGb2UJ/z7Nb
yCvywU0LzPJQwmao5fViNvbjHJjCh4/enHIs1hYE/t6Dvs/lsp5mvWP/egkMb+3D4lQE+cjuUlD5
pAACZcE9e3JwiY8sB63w6RHkAHI42wjutgvemSCAseouCIu4uPVv23JHKuLftQJGaNUIDQxZgXWt
RO03aTG7yiCbbvhiNN9gh76CFa5NBMJAPg8mZVBHGhJCMZl8Ppvg2nZ5jmtqIea20EMdRwTZGNQ+
/b783ncADkiWVbiJaKeAIJVH2TMAEN11hLtnMYsQfVoTciljgCOYsOmMwdf0HoI+SjpRLAOGAOzl
INUzRIPp7+QdMRmycbhKx1na7t3jZVkUadKUVeBGN4zz7tYYZ25ZhQRsfyI8aPNB6fc4jEDkHdVd
OHz03BE4T5oRxCO2qz0LEc1cSneyBiJ5efaK5BISr7RWkFyjVcOrjMYclBLvoYxCvbK7+VRYc89l
1KuHG+TrIJesW0MLjxUDtSspWPIxrtehA5ZqfMbiYBAF2SSdDoJHCJ4H+3eA3EK6oM0Lui6ub6C2
yWoOKK921ucTQBeR0NAtEuYVOlBq7hHFCJQuKvcVQDFs9HqaMl93xK4V21FUs2TD0zsiueNcl/8B
ZVwjijPHYTQ3omt0Vf+hTf+hN52HtoEj27uUNRGRsDdSX81tc3JBElds/0P8YThYxvcXQ/YK8aia
h99Eh4liYv1eJktF2KhgPjTmfEv/lzULUZoV0i2GavGEGpCETI8mnoNFNkv/lZbQI4m5rHh9STpc
VuWetNI7FHQ+YIxzSKqKAdgB4GRloWIhaGoFCWPIx3UHYBgkaLao+8rJzYxssjDzsRXyz7zE6XX4
qmQK7tHqdbAXDtn27gtH5Nhn4aptZEcLn4Qs3MiquRzFbJdNGCKJv4u2FJub3YjCM5lMXk3S1Wly
aUkGr1gQV123G2HWCUcbvmER4uE4zusLxLJ3uDkCInBzgPF+e9NnGMS3k1djLE1YonJMZPrDRQWd
gGdRx2h3unHCsQitMg7fv48uxOeMQU+fdpRmXsUSjtlny7SrhVW6GsdUsaQoi991V7c9iKWojlWJ
ZQXoLZ1BuVsXGkERZaa6hmznL9BBItOewjc4NuYYSxm24ATZsLqZSbl03BZfL5oxLSBAp4JyKPDx
ZzjAIEt0LCNPOZVZBmcTOGyJrqjvYdiXBrr9+SVF+XtFuqab9PZTAxl9ftXNZUNeBFPpixrPRWyd
n3DRcs1cjGHDguwmpOF7f6jGusFP6AJPKyhzAvPP2T9J5gBg05ZlKTkcfataye2nWPHGCgUZLcRX
nARKI8Qsm3AQ2hhbGgmP5y9bGzl4WhzRc7i6gObVvjuksMySvgAls3TVQEqXNXflqp6N2Gw/BX2d
55pg9OoQfYLuwyL6RNu4uGInGhBTQWhyRZEeZ3HnJw3cBAycYCoYzAErAWCpj4Xag+C+hZdp0JbZ
7HoAfygXjJrYgiCT12LMrb07KK9KPvA2T+4crHCWiF8IVLJWF7DQuMdmwkcy4nMN8ETG6uA/dc+8
dnDrQunJFK0nZAbB7rGDDH7/aUSkenqjC26CXvKli69lAQ54zak8M8XritfaXgJhJylBJ14wcIJu
3AIDJwoMiKbKDgjMgQEnJAw4OcAAqC54FGGdDzvZdxjAm1e00eCvCOe1loaIA1xhxRDFVgPo9zhg
1j7CitboVXmsh8GshAwfAJhx/HCTQyTox22ATOOJLIjjFRkaY23vmEAwK7gaqYlUVfcjAKfoZy0z
cgo5we6FFawIOTbqFNdqTw5Vm0du/24Fr1paxSNQnqao3SGHtoPzCKf4UA5t2FCSiOwq+8os2FEH
KJ5fJp3IPjqU21VDPDQNBKgG1hxd1UDHbqsGRGGPrBrEQYOXyeDwozqnbpQaohqvhnI7U42gRk5d
ESIsk7SNL0sSLYOuqiAPcGIE866W7ZgShVyNC0QZQj+RwqiAOLZpSkbYUxVhb5Nqn5Ix9lTF2Gaa
PTjDvpfaG273pm5gvpeU28LutXJZU/TmB1OmTtSzBzZgu2T/U6ltQmPW47UQWsRraw8aE+GinXZl
75xw0TqWhwV+9eyhV3Uqb4o4zvaHQWjE9Idreoi9py5Ywyr2mVi9DtDzQZ8l8o7wngDiI9ipMHyM
CQU4oEy95n56dN8TeWGbvMSDfOQw7071ML1guFgcwfMfjtMGZ0zez0sb6n5w8cb7AQN45kZwnB/k
tLK5lkCeBQE0lVTZY2mEQiQtyxS/+h7DIyVlzyQzKN2nchp7LJleP6AlkzLOXmewftirl8YWyjvD
vLFhVxnIEW9lZO1lgdexeqnvHeEl/CO44oPxgwc0yvNBxmmsbdxyfyTN+S3SzGbdJTEeS3o4Wycf
h8j74Q5n64wYTfc1P8HZOjiq5xytc+LBw8m6u4Z1c66pvfKHOWIxzcmZnpF5zTPIa4bXUe3chu8R
wfZZrAphrHnKyjlH85VdxSMUuMe6dsCzXlWuPgvvdois+8KDzgEYfYwuRzHVZAdxZy5Hw49LCTEz
WWVzsxtReAGKmRlzokc5WXfmdsuwDHI3Ue22OV6yf9jNEXDAH72fB0XvurjlyNU5WV05h+pK21KE
yxVqNxaIWrj6HI1UN0V1zAfnSct8hBExyB6aObn2YcZzrGFY2nkO2i/q5wE1qm7E6KOg66xWmzVo
bVx5IbObcGjr4eUFfgXHQQrr9HwrgdkkSC2CRSPdXiloJozq8ysZy0SA7gpjVk32ehk989EITABk
HOEBCPzUJ+qyzuxvAieoEo4IRSNUfEtbohGhjuFG56U7swge5iH28HCW4Gm8GRO0F6wPYMyVPtwo
5GrqA/dFzgdStyS+Nb9mBNHvpfIpi4ZKQugPWueW/hzAIPsJlNFrRYMvVn+YB7lBKOS01AhjR3gg
9tRneXog+KkBjlqxY/2RegC0Nq2QxH2B5/o9Rwq0hBwOHm48CKbsstcMs/52tMJmnpWZ6H6PuI/g
99iEvNumm/3EWjve0ncZtn6/CYuxWorZtWy1cnIzAmdfvc4e6g6tZi0qEwDTT2UsLPajMs+G6OBU
a+Yr7PFvUTkoWWfkvAhCy2Dl+aFGocRWBdDp8Q+Xz6BsoU7qvBbCSW5bVH2QZ9xjQqoluNMQI7wu
HRTBjxuSZl49mKV3Qqirld8n+9POxYzpPBEn3Yu0KI+mGKFZL/GbU9zNmVkvO7nkYPEKkIr6tUME
WNqIvapu0AgeZ0mZgnEQ4jONTd8+J94RwHCIZV8bsHns5yBZWAYv7B8UoIesbUOBbUGWpHdLqDm8
b08ovWrvRi+xrYMe592tMc7cMnv9HikqXreJqhrRx1gmtVgiOEGwhLwjXL35jmBAhJJq4eHvaUc4
DsLJGI2IPfT4+RCa7l5+seYXeoiXVWKIySXKonrp3h5OC+bmDilTuPv2hwbunBQZoNNmkXn2QhoU
ZmAwt+9C6m1TNASF0g2vNm4Y51VJY5yply9SMB38ieev9l3aDiZR/rK3bBw1HcBWfpvA6QzZdcDd
wpcJRGrmNA+9o75myEPvqCFaQnCYnPnIxdwy/5HB3f8uA6iP2zyK/Us6fjx0j3brHlVeqt3NFI3p
AvP4LCRBhnGBZgzLtXdryqswMDIrGU4/z7qGJju1Bw+bD6ivIFjss/Ju7YTdV049EN9fszCy6W7b
gHrsvU0b9nD2du1pD9tu1GPlH47rnymp3m2HLAEM1gT6x0GBvi44ELKFTUst2SIqGuGyhdkLzFa0
8fYxmqturHPsCOcLJxNMIDMlgYQMsoxmNs+xvcfYvmJTEh5H/kJ7SSP1vHSKPcjbTfRu8xwet45D
zczz86sUymjahuay98Dst5yI4KB/vyUqgauc8Oh4KgUND0Y/GP40miAOosRT0M1EIX06r5/G8D/P
noPO+jDdeb8lN/G6GiUtvdRF8p1zsfjgKp2/+wsAAP//AwBQSwMEFAAGAAgAAAAhAGxXJQDZCAAA
qVUAAB8AAABjbGlwYm9hcmQvZGlhZ3JhbXMvZHJhd2luZzEueG1s7Fxbb+M2Fn5fYP+DoHdOKFKU
xGCcQpKttkC6DZoMuq+MLNtCdVtJySRd7H/fQ1KyZSfTXDrJJBjmwabpQ/rwdvid8x3l4w83ZWFd
Z22X19XMdj5g28qqtF7m1Xpmf7pIUGBbXS+qpSjqKpvZt1ln/3Dyz398XHbN8bIVn0HQgj6q7ni5
Lmf2pu+b46OjLt1kpeg+1E1Wwberui1FDx/b9dHQqCyOCMbe0TIX61aU9thJ19zppMzTtu7qVf8h
rcujerXK02zsRnYSHHYi7nTxsB6lyCv7RA2ray7aLNPl6vrHtjlvzlr9Mf3X9Vlr5cuZDRNViRJm
xD7afjWIQsWRnJ6Dtmvd0SDeNVZZL7PiZ+jrvxwH7sJxIhQx4qAFdikKKI+RQwOcRIswYjz636Bd
df1YdaTcTpddq04NRxzfrNry5KM4hhm1bmY24Qx7lNnWLWwEBtPqYzk2cZzd9FYqBTyMXde2Uing
upwHjhQ42vWUXnX9j1mtehXXp12v2q+XY0lsxlJ6U43FNkt7q1Az2qvXVr1eylfVvhG9bCdVkUX5
XtbX2UWtanqpO6wGKKXkQZ3dt0V1V2pf9VFifG9Uf+NQ74xUi8FvaE2GgtIOytPxV3WSF4UaQFFZ
n2c2zC2omQrY4KtC9FAsG1j8rlrblijWcPzSvoVFFsddXeRL2Vp9kEcpi4vWuhYwSyJNs6p3tNxG
LDNdHWD409O1ucp+hRVV0kNVJ/rDquKqPKwSRbMR+5UwJnWU5e+rld5TrWm7fi66jf4t9ZVWocz7
rLWKHAyCUmxcmKKSI8pWK1hyvfzqoOgNKY9M198WcPLEcVH9lq3goMHaDmNN2/WlnAW9O8A6wWSO
ewTUVA1kyxXM27Yt1vP0QNuhyU63J7bXA4JG6vfrqt+2L/OqbtUhmYxuGPNobCZ2IEh46DPiI89h
EQqpyxD3WIiiIPIWIQnCMCRvyw4wH2M/UCN8Z2ZgT/Px9I/vd6zAPdLD2VfHYmuijBEA02WMwOpv
GAHG50kUxg4K55EyAhRFceggEuGAzwOXLzz68kbAc33H1WDAI5h7vl7UEQtw7nGXaCwAlyVh9D3a
AIdih3iD5uPZH9+1DRgHKpHAk6T3ZkX3+fYsBiA6Axv0tf2OYYNLF7EbMI4C1w0RwwAbopDGKA7j
BBPKE5d4L24xOPN8CXDlQTHeg3KUjPdgvAfpQP215/LVvAeHxyGmgYN8ngQomrschck8Ql4URm4U
Bj6c0DdlBvZA9RiNmPrQbzaGsKf5CBjGd+M8TIIPgHlMBOH1bABjEEPgrouSBYe4gesGiCcMrAFj
4EPQiETRy0MBx2OYYKqwwPtyHiZof4xZfSmYOBF9hGMwAKOHPQgtaJwHHfM0MccvxTu/GmoIGI1o
5McowWyOGPgKKPLdOcKMYBKBMxGH/MVRg+MHmBKuvQfqHAQbHJ9Sjw7RBs+BkOh4OEfQICPRkniw
ZGFmS0pBRX5HACE970FExmiBcVLxdaut+9/zfnO+EQ1QOTpavO4GsmHdWU3djdXbW0xfbtM4/NeM
uMN1Kfpf6qX+FQfTrYMOEftJPfBeQ3C/z6teS3N3qLxz6a47NexhQOCeQWNZ820G5ajf19TI3qB2
9d2O2lDS44JvFVaWYX9Yg+Crjmu6KJzLaVXD2ltDuVRj/W5YsON17b2LBZXbfVjklSUkH8vk+kIj
q0tFkQF3NM7KuJ/l0DXHsmWfoKNnsS7j/vhr3+V+1oX+TdaFPKr9Qxbwnn1QAHEG6wNzMgRooTwh
ZmSxv4nq5a1sewnvQPZ2TZrkYDtORdefiRa4ZVgAYMzBzGzq9k/b+txKSq/7z5VoM9sqfq7AZHBG
IHbaT8rtpHw5KVdXZVwDr+fAqjapLhJgXmHFU+hfk4LDh7hX5JdUrqrDq75e5ZJhheFoXdXyd/25
JNHUNmxkDbC9VnGtCdYdz2gts9WFuDz/E4hJT0VuVOvqvEllAXQ5A1pWkYhyW487rZgIRJrwmoqO
YlC3+zZcKe52KgdU6LZLqFcSMIyBmG6lBq1UW237rEKfzmF6QFUZWbKtP7JWJivAubKtZd72EGeB
Ulf2cZEJ+GY4bP1JeHEansv5AQXgVfV7+VV6l3OuO/paup7OwH5Znw60fYqy90zADFs/fZp57KDX
rFrKrfzbE6dY7zXYVOrQjCdFn5p/7ycTPO9Cn3b1ZZaUhkCBsDhGcwqIJcGuh4K5TxFeeE7oYhIu
8OLFEYsfUBp4CrAEgUPIsOdGfsRAFmVO9m53A1nAahjIoq2IgSz3cMTq5t3P9zGQZZsa9V1AlrPN
bZen3cGF/XZBiwuYZcYkyiCvjzKeeQk/DmbMF3M294FQBWbVQUngxsCvcgx5GIz7IefE48mLwwyH
OJ7Ph6RM33Mc4moPcws0qE+gWmdimNiI9KQg9qydC8D9JjaiY1gmNqLwqMo/NbER2wCN4+89NvJT
vl7vwwzwTvSkyN3xHUeNTmdEghpHoRp8gMReIXTy3Cv/cbAm5BS7BC8Qxwsu+Z4YhcSDYAojUejy
cMEwe1VYQzwXruxDysfAGjiEU3bBwBoVwzXxExM/+eKDNgbWfPew5jwvr+DpOnic9eDifrshFEeh
DcXUfFu08aSb+HFogy2cEKImHEWOBw+1Qpo6iqIoQSFL5ozCte94L59dQnzfJz7kjwBRaMiavdwE
k18C8SL5N+QFmPwSSaqb/BLwhU1+ickveSC/5NcmaxXW2A+kbHNDnphfcZjC8hJJJp4EGwpr0Nfn
a557ET8Oa5B54s1ZwJCbRBi5DuSFwDOqTD4EE7gRpIUkNH7xyAYlDvYCsKHy4a97CRuTzGoiG9qH
N2DDgI0BdxqwYcDGA2Dj97r9Y1XUn99JXMM6nVGJNeAFkkNeP7Dx7Iv4sWDDhyRUylAc+xget4eH
bSMaccTh/qdz+LddAEJeFWzcH7wxYMOADQM2zJMze/8rzYANAzYeAhvZ5TvBGQPM+EbJGlOU8aQb
+F6UMTzzov/9qPow/LfUk/8DAAD//wMAUEsBAi0AFAAGAAgAAAAhAGiBMdJZAQAA0QQAABMAAAAA
AAAAAAAAAAAAAAAAAFtDb250ZW50X1R5cGVzXS54bWxQSwECLQAUAAYACAAAACEArTA/8cEAAAAy
AQAACwAAAAAAAAAAAAAAAACKAQAAX3JlbHMvLnJlbHNQSwECLQAUAAYACAAAACEAtGfdfIwaAABK
nQAAHAAAAAAAAAAAAAAAAAB0AgAAY2xpcGJvYXJkL2RpYWdyYW1zL2RhdGExLnhtbFBLAQItABQA
BgAIAAAAIQBGAF7ajwEAAEgEAAAfAAAAAAAAAAAAAAAAADodAABjbGlwYm9hcmQvZHJhd2luZ3Mv
ZHJhd2luZzEueG1sUEsBAi0AFAAGAAgAAAAhAGjmN9gyAQAA/AMAACoAAAAAAAAAAAAAAAAABh8A
AGNsaXBib2FyZC9kcmF3aW5ncy9fcmVscy9kcmF3aW5nMS54bWwucmVsc1BLAQItABQABgAIAAAA
IQC6I1YXQQYAAA0aAAAaAAAAAAAAAAAAAAAAAIAgAABjbGlwYm9hcmQvdGhlbWUvdGhlbWUxLnht
bFBLAQItABQABgAIAAAAIQADwKI8EQQAAN9BAAAeAAAAAAAAAAAAAAAAAPkmAABjbGlwYm9hcmQv
ZGlhZ3JhbXMvY29sb3JzMS54bWxQSwECLQAUAAYACAAAACEAa5uWFBgEAAANUQAAIgAAAAAAAAAA
AAAAAABGKwAAY2xpcGJvYXJkL2RpYWdyYW1zL3F1aWNrU3R5bGUxLnhtbFBLAQItABQABgAIAAAA
IQCtkm1maRAAAPeWAAAeAAAAAAAAAAAAAAAAAJ4vAABjbGlwYm9hcmQvZGlhZ3JhbXMvbGF5b3V0
MS54bWxQSwECLQAUAAYACAAAACEAbFclANkIAACpVQAAHwAAAAAAAAAAAAAAAABDQAAAY2xpcGJv
YXJkL2RpYWdyYW1zL2RyYXdpbmcxLnhtbFBLBQYAAAAACgAKAOYCAABZSQAAAAA=
">
<v:imagedata src="file:////Users/timbell/Library/Group%20Containers/UBF8T346G9.Office/msoclip1/01/clip_image001.png"
o:title="" cropleft="-12095f" cropright="-11867f"/>
<o:lock v:ext="edit" aspectratio="f"/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--></span><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The following rules would be applied<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l1 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
quota administrator would not be able to increase the limit of a child project
such that the sum of the limits of the child project exceeds the parent.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
ATLAS administrator could not increase the limit for either Physics or
Operations<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
Physics administrator could increase the limit for either the Higgs or Simulation
project by 10<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
Operations administrator could not increase the limit for either Workflow or
Web<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l1 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
quota administrator cannot set the limit on a project such that the limits on
the children exceeds the parent limit<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
ATLAS administrator could not set the Operations limit to be 50 as
Limit(Workflow)+Limit(Web)>Limit(Operations)<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l1 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
quota administrator can set the limit below the usage<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">The
Physics administrator could lower the Simulation limit to 5 even though the used
resources at 10<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l1 level1 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Creating
a new resource requires that the usage in the project is less than or equal to
the limit at all levels of the tree<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">No
additional resources could be created in Simulation since Used(Simulation)>=Limit(Simulation)<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">No
additional resources could be created in Higgs as </span><span style="font-family: "Times New Roman",serif; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-GB;">HierarchicalUsage</span><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">(Physics)>=Limit(Physics).
The error message for this case would need to indicate the quota limit is in
Physics. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 72.0pt; mso-add-space: auto; mso-list: l1 level2 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: "Courier New",serif; mso-fareast-font-family: "Courier New"; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">o<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Up to 25
new resources could be created in Web since Usage(Web)+25<=Limit(Web), HierarchicalUsage(Operations)+25<=Limit(Operations)
and HierarchicalUsage(ATLAS)+25<=Limit(ATLAS). After this operation,<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 108.0pt; mso-add-space: auto; mso-list: l1 level3 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Wingdings; mso-bidi-font-family: Wingdings; mso-fareast-font-family: Wingdings; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">§<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Used(Web)=30<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 108.0pt; mso-add-space: auto; mso-list: l1 level3 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Wingdings; mso-bidi-font-family: Wingdings; mso-fareast-font-family: Wingdings; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">§<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">HierarchicalUsage(Operations)=60<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: 108.0pt; mso-add-space: auto; mso-list: l1 level3 lfo4; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="font-family: Wingdings; mso-bidi-font-family: Wingdings; mso-fareast-font-family: Wingdings; mso-fareast-language: EN-GB;"><span style="mso-list: Ignore;">§<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">HierarchicalUsage(ATLAS)=80<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
<o:PixelsPerInch>96</o:PixelsPerInch>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves>false</w:TrackMoves>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-GB</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="382">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 9"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:fixed;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;
mso-font-charset:2;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:0 268435456 0 0 -2147483648 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:1;
mso-generic-font-family:roman;
mso-font-format:other;
mso-font-pitch:variable;
mso-font-signature:0 0 0 0 0 0;}
@font-face
{font-family:"Calibri Light";
panose-1:2 15 3 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-1610611985 1073750139 0 0 415 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073786111 1 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
h1
{mso-style-priority:9;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-link:"Heading 1 Char";
mso-style-next:Normal;
margin-top:12.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan lines-together;
page-break-after:avoid;
mso-outline-level:1;
font-size:16.0pt;
font-family:"Calibri Light",sans-serif;
mso-ascii-font-family:"Calibri Light";
mso-ascii-theme-font:major-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:major-fareast;
mso-hansi-font-family:"Calibri Light";
mso-hansi-theme-font:major-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:major-bidi;
color:#2F5496;
mso-themecolor:accent1;
mso-themeshade:191;
mso-font-kerning:0pt;
mso-fareast-language:EN-US;
font-weight:normal;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-unhide:no;
mso-style-locked:yes;
mso-style-link:"Heading 1";
mso-ansi-font-size:16.0pt;
mso-bidi-font-size:16.0pt;
font-family:"Calibri Light",sans-serif;
mso-ascii-font-family:"Calibri Light";
mso-ascii-theme-font:major-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:major-fareast;
mso-hansi-font-family:"Calibri Light";
mso-hansi-theme-font:major-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:major-bidi;
color:#2F5496;
mso-themecolor:accent1;
mso-themeshade:191;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:601913043;
mso-list-template-ids:-1954912558;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1046291442;
mso-list-type:hybrid;
mso-list-template-ids:1811307082 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New",serif;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:1760254847;
mso-list-template-ids:-1372827998;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New",serif;
mso-bidi-font-family:"Times New Roman";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3
{mso-list-id:2107576289;
mso-list-template-ids:-1102252324;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-fareast-language:EN-US;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--><br />
<div class="MsoNormal">
<span style="font-family: "Times New Roman",serif; mso-fareast-language: EN-GB;">Based on past experience with Nova quotas, the aim would be to calculate
all the usages (both Used and HierarchicalUsage dynamically at resource
creation time). The calculation of the hierarchical usage could be expensive
however since it requires the navigation of the whole tree for each resource
creation. Some additional Keystone calls to get the entire project tree would
be needed. This may limit the use in large scale environments but the
performance would only be affected where the functionality was used.<o:p></o:p></span></div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com4tag:blogger.com,1999:blog-6032896665180559.post-4914916607993483872017-06-09T05:55:00.000-07:002017-06-09T09:07:24.635-07:00Experiences with Cinder in ProductionThe CERN OpenStack cloud service is providing block storage via Cinder since Havana days in early 2014. Users can choose from seven different volume types, which offer different physical locations, different power feeds, and different performance characteristics. All volumes are backed by Ceph, deployed in three separate clusters across two data centres.<br />
<br />
Due to its flexibility, the volume concept has become very popular with users and the service has hence grown during the past years to over 1PB of allocated quota, hosted in more than 4'000 volumes. In this post, we'd like to share some of the features we use and point out some of the pitfalls we've run into when running (a very stable and easy to maintain) Cinder service in production.<br />
<br />
<h3>
Avoiding sadness: Understanding the 'host' parameter</h3>
<div>
<br /></div>
<div>
With the intent to increase the resiliency, we configured the service from the start to run on multiple hosts. The three controller nodes were set up in an identical way, so all of them ran the API ('<span style="font-family: "courier new" , "courier" , monospace;">c-api</span>'), scheduler ('<span style="font-family: "courier new" , "courier" , monospace;">c-sched</span>') and volume ('<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>') services.</div>
<div>
<br /></div>
<div>
With the first upgrades, however, we realised that there was a coupling between a volume and the '<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>' service that had created it: each volume is associated with its creation host which, by default, is identified by hostname of the controller. So, when the first controller needed to be replaced, the '<span style="font-family: "courier new" , "courier" , monospace;">c-sched</span>' wasn't able to find the original '<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>' service which would be able to execute volume operations. At the time, we fixed this by changing the corresponding volume entries in the Cinder database to point to the new host that was added.</div>
<div>
<br /></div>
<div>
As the Cinder configuration allows the 'host' to be set directly in '<span style="font-family: "courier new" , "courier" , monospace;">cinder.conf</span>', we set this parameter to be the same on all controllers with the idea to remove the coupling between the volume and the specific '<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>' which was used to create it. We ran like this for quite a while and although we never saw direct issues related to this setting, in hindsight it may explain some of the issues we had with volumes getting stuck in transitional states. The main problem here is the clean-up being done as the daemons start up: as they assume exclusive access to 'their' volumes, volumes in transient states will be "cleaned up", e.g. their state reset, when a daemon starts, so in a setup with identical 'host's, this may cause undesired interferences.</div>
<div>
<br /></div>
<div>
Taking this into account, our setup has been changed to keep the '<span style="font-family: "courier new" , "courier" , monospace;">c-api</span>' on all three controller, but run the '<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>' and '<span style="font-family: "courier new" , "courier" , monospace;">c-sched</span>' services <i>on one host only</i>. Closely following the recent work of the Cinder team to improve the locking and allow for Active/Active HA we're looking forward to have Active/Active HA '<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>' services fully available again.</div>
<div>
<br /></div>
<h3>
Using multiple volume types: QoS and quota classes</h3>
<br />
The scarce resource on our Ceph backend is not space, but IOPS, and after we handed out the first volumes to users, we quickly realised that some resource management was needed. We achieved this by creating a QoS spec and associating it with the one volume type we had at the time:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder qos-create std-iops write_iops_sec=100 read_iops_sec=100</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder qos-associate <std_iops_qos_id> <std_volume_type_id></span><br />
<br />
This setting does not only allow you to limit the amount of IOPS used on this volume type, but also to define different service levels. For instance, for more demanding use cases we added a high IOPS volume type to which access is granted on a per request basis:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder type-create high-iops</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder qos-create high-iops write_iops_sec=500 read_iops_sec=500</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder qos-associate <high_iops_qos_id> <high_iops_volume_type_id></span><br />
<br />
Note that both types are provided by the <i>same backend and physical hardware</i> (which also allows for a conversion without data movement between these types using '<span style="font-family: "courier new" , "courier" , monospace;">cinder retype</span>')! Note also that for attached volumes a detach/re-attach cycle is needed to have QoS changes taking effect.<br />
<br />
In order to manage the initial default quotas for these two (and the other five volume types the service offers), we use Cinder's support for <i>quota classes</i>. As apart from the std-iops volume type all other volume types are only available on request, the initial quota is usually set to '0'. So, in order to create the default quotas for a new type, we would hence update the default quota class by running a command like:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder type-create new-type</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># cinder quota-class-update --volume-type new-type --volumes 0 --snapshots 0 --gigabytes 0 default</span><br />
<br />
Of course, this method can also be used to define different initial quotas for new volume types, but it is in any case a way to avoid setting the initial quotas explicitly after project creation.<br />
<h3>
</h3>
<h3>
<br /></h3>
<h3>
Fixing a long-standing issue: Request timeouts and DB deadlocks</h3>
<br />
For quite some time, our Cinder deployment had suffered from request timeouts leading to volumes left in error states when doing parallel deletions. Though easily reproducible, this was infrequent (and subsequently received the corresponding attention ...). Recently, however, this became a much more severe issue with the increased use of Magnum and Kubernetes clusters (which use volumes and hence launch parallel volumes deletions at larger scale when being removed). This affected the overall service availability (and, subsequently, received the corresponding attention here as well ...).<br />
<br />
In this situations, the '<span style="font-family: "courier new" , "courier" , monospace;">c-vol</span>' logs showed lines like<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="background-color: white; color: #333333; font-size: 12px;">"Deadlock detected when running 'reservation_commit': Retrying ..."</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="background-color: white; color: #333333; font-size: 12px;"><br /></span></span>
and hence indicated locking problem. We weren't able to pinpoint in the code how a deadlock would occur, though. A first change that mitigated the situation was to reduce the '<span style="font-family: "courier new" , "courier" , monospace;">innodb_lock_wait_timeout</span>' from its default value of 50 seconds to 1 second: the client was less patient and exercised the retry logic decorates the database interactions much earlier. Clearly, this did not address the underlying problem, but at least allowed the service to handle these parallel deletions in a much better way.<br />
<br />
The real fix, suggested by a community member, implied to try and change a setting we had carried forward since the initial setup of the service: the connection string in '<span style="font-family: "courier new" , "courier" , monospace;">cinder.conf</span>' was not specifying a driver and hence using the mysql Python wrapper (rather than the recommended '<span style="font-family: "courier new" , "courier" , monospace;">pymysql</span>' Python implementation). After changing our connection from<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">connection = mysql://cinder:<pw>@<host>:<port>/cinder</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
to<br />
<br class="Apple-interchange-newline" />
<span style="font-family: "courier new" , "courier" , monospace;">connection = mysql+pymysql://cinder:<pw>@<host>:<port>/cinder</span><br />
<br />
the problem basically disappeared!<br />
<br />
So the underlying reason was the management of the green thread parallelism in the wrapper vs. the native Python implementation: while the former enforces serialisation (and hence eventually deadlocks in SQLAlchemy), the latter allows for proper parallel execution of the requests to the database. The OpenStack oslo team is now looking into issuing a warning when it detects this obsolete setting.<br />
<br />
As using the '<span style="font-family: "courier new" , "courier" , monospace;">pymysql</span>' driver is generally recommended and, for instance, default in devstack deployments, volunteers to help with this issue had a really hard time to reproduce the issues we experienced ... another lesson learnt when keeping services running for a longer period :)<br />
<br />
<br />Arne Wiebalckhttp://www.blogger.com/profile/08776564148597522650noreply@blogger.com6tag:blogger.com,1999:blog-6032896665180559.post-7664979117589734752017-06-06T10:30:00.005-07:002017-06-07T03:02:09.330-07:00OpenStack papers community on Zenodo<br />
At the recent summit in Boston, Doug Hellmann and I were discussing research around OpenStack, both the software itself but also how it is used by applications. There are many papers being published in proceedings of conferences and PhD theses but finding out about these can be difficult. While these papers may not necessarily lead to open source code contribution, the results of this research is a valuable resource for the community.<br />
<br />
Increasingly, publications are made with <a href="https://en.wikipedia.org/wiki/Open_access">Open Access</a> conditions which are free of all restrictions on access. For example, all projects receiving European Union Horizon 2020 funding are required to make sure that any peer-reviewed journal article they publish is openly accessible, free of charge. Reviewing with the OpenStack scientific working group, Open access was also felt to be consistent with OpenStack's <a href="https://wiki.openstack.org/wiki/Open">Open principles</a> of Open Source, Open Design, Open Development and Open Community.<br />
<br />
There are a number of different repositories available where publications such as this can be made available. The OpenStack scientific working group are evaluating potential approaches and <a href="http://zenodo.org/">Zenodo</a> looks like a good candidate as it is already widely used in the research community, <a href="https://github.com/zenodo">open source</a> on github and the application also runs in the CERN Data Centre on OpenStack. Preservation of data is one of CERN's key missions and this is included in the service delivery for Zenodo.<br />
<br />
The name Zenodo is derived from Zenodotus, the first librarian of the Ancient Library of Alexandria and father of the first recorded use of metadata, a landmark in library history.<br />
<h2>
Accessing the Repository</h2>
<div>
The list of papers can be seen at <a href="https://zenodo.org/communities/openstack-papers">https://zenodo.org/communities/openstack-papers</a>. Along with keywords, there is a dedicated search facility is available within the community so that relevant papers can be found quickly.</div>
<h2>
Submitting New Papers</h2>
<div>
Zenodo allows new papers to be submitted for inclusion into the OpenStack Papers repository. There are a number of steps to be performed.<br />
<br />
Please ensure that these papers are available under open access conditions before submitting them to the repository if published elsewhere. Alternatively, if the papers can be published freely, they can be published in Zenodo for the first time and receive the DOI directly.</div>
<div>
<ol>
<li>Log in to <a href="http://zenodo.org/">Zenodo</a>. This can be done using your github account if you have one or by registering a new account via the 'Sign Up' button.</li>
<li>Once logged in, you can go to the openstack repository at <a href="https://zenodo.org/communities/openstack-papers">https://zenodo.org/communities/openstack-papers</a> and upload a new paper.</li>
<li>The submission will then be verified before publishing.</li>
</ol>
<div>
To submit for this repository, you need to provide</div>
<div>
<ul>
<li>Title of the paper</li>
<li>Author list</li>
<li>Description (the Abstract is often a good content)</li>
<li>Date of publication</li>
</ul>
<div>
If you know the information, please provide the following also</div>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Digital_object_identifier">DOI</a> (Digital Object Identifier) used to uniquely identify the object. In general, these will already be allocated to the paper since the original publication will have allocated one. If none is specified, Zenodo will create one, which is good for new publications but bad practice to generate duplicate DOIs for published works. So please try to find the original, which also it helps with future cross referencing.</li>
<li>There are optional fields at upload time for adding more metadata (to make it machine readable), such as “Journal” and “Conference”. Adding journal information improves the searching and collating of documents for the future so if this information is known, it is good to enter it.</li>
</ul>
<div>
Zenodo provides the synchronisation facilities for repositories to exchange information (<a href="https://zenodo.org/oai2d?verb=ListRecords&set=user-openstack-papers&metadataPrefix=oai_dc">OAI 2.0</a>). Planet OpenStack feeds using this would be an interesting enhancement to consider or adding RSS support to Zenodo would be welcome contributions.</div>
</div>
<div>
<br /></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<br />
<br />
<h2>
</h2>
<div>
<div style="font-family: -webkit-standard; text-size-adjust: auto;">
</div>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com3tag:blogger.com,1999:blog-6032896665180559.post-82175559633470350662017-05-02T04:21:00.002-07:002017-05-02T04:21:55.038-07:00Migrating to Keystone Fernet Tokens<div style="text-align: justify;">
<span lang="ES" style="text-align: justify;">For several OpenStack releases, the Identity service in OpenStack offers an additional token format
than UUID, called Fernet. </span><span style="text-align: justify;">This token format
has a series of advantages over the UUID, the most prominent for us is that it
doesn't need to be persisted. We were also interested in a speedup of the token creation and validation.</span></div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">At CERN, we have
been using the UUID format for the tokens since the beginning of the cloud
deployment in 2013. Normally in the keystone database </span>we have around
300,000 tokens with an expiration of 1 day. In order to keep the database size
controlled, we run the token_flush procedure every hour.</div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">In the Mitaka
release, all remaining bugs were sorted out and since the Ocata release of
OpenStack, Fernet is now the default. </span>Right now, we are
running keystone in Mitaka and we decided to migrate to the Fernet token format
before the upgrade of the Ocata release.</div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">Before reviewing the upgrade from UUID to Fernet, let's have a brief look on the architecture of the identity
service. The service resolves into a set of load balancers </span>and then they
redirect to a set of backends running keystone under apache. This allows us to
replace/increase the backends transparently.</div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="ES">The very first
question is how many keys we would need. If we take the formula from [1]:</span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="ES"><i>fernet_keys =
validity / rotation_time + 2</i><o:p></o:p></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="ES">If we have a
validity of 24 hours and a rotation every 6 hours, we would need 24/6 + 2 = 6 keys<o:p></o:p></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">As we have
several backends, the first task is to distribute the keys among the backends,
for that we are using puppet that provides the secrets </span>in the /etc/keystone/fernet-keys
folder. With that we ensure that a new introduced backend will always have the
last set of keys available.</div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">The second task
is how to rotate them. In our case we are using a cronjob in our rundeck
installation that rotates the secrets and introduces a new one. </span>This job is doing
exactly the same as the keystone-manage token-flush command. One important
aspect is that on each rotation, you need to reload or restart the Apache daemon
to load the keys from the disk.</div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">So we prepare all
this changes in the production setup quite some time ago, and we were testing
that the keys were correctly updated and distributed. </span>On April 5th, we
decided to go on. This is the picture of the API messages during the
intervention.</div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-kxz-gw1qpMY/WQhNHpVQqfI/AAAAAAAAEbI/jsIHe7gcPuQ3KmgBwdyXFppbi_wd6PgaQCLcB/s1600/fernet_token_1.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="110" src="https://4.bp.blogspot.com/-kxz-gw1qpMY/WQhNHpVQqfI/AAAAAAAAEbI/jsIHe7gcPuQ3KmgBwdyXFppbi_wd6PgaQCLcB/s400/fernet_token_1.JPG" width="400" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">There
are two distinct steps in the API messages usage. The first one is a peak of invalid tokens. This is triggered by
the end users trying to validate UUID tokens after the change. </span>The second peak
is related to OpenStack services that are using trusts, like Magnum and Heat.
From our past experience, these services can be affected by a massive invalidation of
tokens. The trust credentials are cached and you need to restart both
services so both services will get their Fernet tokens.</div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">Below is a picture
of the size of the token table in the keystone database, as we are now in Fernet is going slowly down to zero, due to the hourly token_flush command </span>I mentioned
earlier.</div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-aWXQRs703Yw/WQhNKjUO9OI/AAAAAAAAEbQ/grni-du5r8EEio-dyYxOVaUH2l1EoZDBQCLcB/s1600/fernet_token_2.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="105" src="https://3.bp.blogspot.com/-aWXQRs703Yw/WQhNKjUO9OI/AAAAAAAAEbQ/grni-du5r8EEio-dyYxOVaUH2l1EoZDBQCLcB/s400/fernet_token_2.JPG" width="400" /></a></div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">The last picture
is the response time of the identity service during the change. As you can see
the response time is better than on UUID as stated in [2]<o:p></o:p></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="ES"><br /></span></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-qPzX8AB-MBM/WQhNKqeGT-I/AAAAAAAAEbM/-D_vZvt3xhEPPVKjMws8uisPMN0pD-ZsACLcB/s1600/fernet_token_3.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="105" src="https://3.bp.blogspot.com/-qPzX8AB-MBM/WQhNKqeGT-I/AAAAAAAAEbM/-D_vZvt3xhEPPVKjMws8uisPMN0pD-ZsACLcB/s400/fernet_token_3.JPG" width="400" /></a></div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="text-align: justify;">
<div style="text-align: justify;">
<span lang="ES">In the Ocata
release, more improvements are on the way to improve the response time, and we
are working to update the identity service in the near future.<o:p></o:p></span></div>
</div>
<br />
<h2>
<span lang="ES">References:</span></h2>
<div class="MsoNormal">
</div>
<ol>
<li>Fernet token FAQ at <a href="https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html">https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html</a></li>
<li>Fernet token performance at <a href="http://blog.dolphm.com/openstack-keystone-fernet-tokens/">http://blog.dolphm.com/openstack-keystone-fernet-tokens/</a></li>
<li>Payloads for Fernet token format at <a href="http://blog.dolphm.com/inside-openstack-keystone-fernet-token-payloads/">http://blog.dolphm.com/inside-openstack-keystone-fernet-token-payloads/</a></li>
<li>Deep dive into Fernet at <a href="https://developer.ibm.com/opentech/2015/11/11/deep-dive-keystone-fernet-tokens/">https://developer.ibm.com/opentech/2015/11/11/deep-dive-keystone-fernet-tokens/</a></li>
</ol>
Anonymoushttp://www.blogger.com/profile/09376084384626279065noreply@blogger.com5tag:blogger.com,1999:blog-6032896665180559.post-64236339985817473292017-02-09T01:29:00.002-08:002017-02-09T23:29:16.692-08:00os_type property for Windows images on KVM <br />
The OpenStack images have a long list of properties which can set to describe the image meta data. The full list is described in the <a href="http://docs.openstack.org/cli-reference/glance-property-keys.html">documentation</a>. This blog reviews some of these settings for Windows guests running on KVM, in particular for Windows 7 and Windows 2008R2.<br />
<br />
At CERN, we've used a number of these properties to help users filter images such as the OS distribution and version but also added some additional properties for specific purposes such as<br />
<br />
<ul>
<li>when the image was released (so the images can be sorted by date)</li>
<li>whether the image is the latest recommended one (such as setting the CentOS 7.2 image to not recommended when CentOS 7.3 comes out)</li>
<li>which CERN support team provided the image </li>
</ul>
<br />
For a typical Windows image, we have<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">$ glance image-show 9e194003-4608-4fe3-b073-00bd2a774a57</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">+-------------------+----------------------------------------------------------------+</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| Property | Value |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">+-------------------+----------------------------------------------------------------+</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| architecture | x86_64 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| checksum | 27f9cf3e1c7342671a7a0978f5ff288d |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| container_format | bare |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| created_at | 2017-01-27T16:08:46Z |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| direct_url | rbd://b4f463a0-c671-43a8-bd36-e40ab8d233d2/images/9e194003-4</span></span><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-variant-ligatures: no-common-ligatures;"> |</span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| disk_format | raw |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| hypervisor_type | qemu |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| id | 9e194003-4608-4fe3-b073-00bd2a774a57 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| min_disk | 40 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| min_ram | 0 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| name | Windows 10 - With Apps [2017-01-27] |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| os | WINDOWS |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| os_distro | Windows |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| os_distro_major | w10entx64 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| os_edition | DESKTOP |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| os_version | UNKNOWN |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| owner | 7380e730-d36c-44dc-aa87-a2522ac5345d |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| protected | False |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| recommended | true |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| release_date | 2017-01-27 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| size | 37580963840 |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| status | active |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| tags | [] |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| updated_at | 2017-01-30T13:56:48Z |</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| upstream_provider | https://cern.service-now.com/service-portal/function.do?name |</span></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-variant-ligatures: no-common-ligatures;">| virtual_size | None |</span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">| visibility | public |</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">+-------------------+----------------------------------------------------------------+</span></div>
<div class="p1">
<span class="s1"><span style="font-family: inherit;"><br /></span></span></div>
Recently, we have seen some cases of Windows guests becoming unavailable with the <a href="https://en.wikipedia.org/wiki/Blue_Screen_of_Death">BSOD</a> error "CLOCK_WATCHDOG_TIMEOUT (101)". On further investigation, these tended to occur around times of heavy load on the hypervisors such as another guest doing CPU intensive work.<br />
<br />
Windows 7 and Windows Server 2008 R2 were the guest OSes where these problems were observed. Later OS levels did not seem to show the same problem.<br />
<br />
We followed the standard processes to make sure the drivers were all updated but the problem still occurred.<br />
<br />
Looking into the root cause, the <a href="https://access.redhat.com/solutions/755943">Red Hat support articles</a> were a significant help.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">"In the environment described above, it is possible that 'CLOCK_WATCHDOG_TIMEOUT (101)' BSOD errors could be due to high load within the guest itself. With virtual guests, tasks may take more time that expected on a physical host. If Windows guests are aware that they are running on top of a Microsoft Hyper-V host, additional measures are taken to ensure that the guest takes this into account, reducing the likelihood of the guest producing a BSOD due to time-outs being triggered."</span><br />
<br />
These suggested to use the <span style="font-family: "courier new" , "courier" , monospace;">os_type</span> parameter to help inform the hypervisor to use some additional flags. However, the OpenStack <a href="http://docs.openstack.org/cli-reference/glance-property-keys.html">documentation</a> explained this was a XenAPI only setting (which would not therefore apply for KVM hypervisors).<br />
<br />
It is not always clear which parameters to set for an OpenStack image. Setting <span style="font-family: "courier new" , "courier" , monospace;">os_distro</span> has a value such <span style="font-family: "times" , "times new roman" , serif;">as 'windows' or 'ubuntu'. While the flavor of the OS could be determined, the </span><span style="font-family: "times" , "times new roman" , serif;"><span style="background-color: white; color: #333333;">setting of</span><span style="background-color: white; color: #333333; font-size: 14px;"> </span></span><span style="background-color: white; color: #333333;"><span style="font-family: "courier new" , "courier" , monospace;">os_type</span><span style="font-family: open sans, helvetica, arial, sans-serif;"> is needed to be used by the code.</span></span><br />
<span style="background-color: white; color: #333333; font-family: "open sans" , "helvetica" , "arial" , sans-serif; font-size: 14px;"><br /></span>
Thus, in order to get the best behaviour for Windows guests, from our experience, we would recommend setting both the <span style="font-family: "courier new" , "courier" , monospace;">os_distro</span> and <span style="font-family: "courier new" , "courier" , monospace;">os_type </span>as follows.<br />
<ul>
<li>os_distro = 'windows'</li>
<li>os_type = 'windows'</li>
</ul>
<div>
When the <span style="font-family: "courier new" , "courier" , monospace;">os_type</span> parameter is set, some additional XML is added to the KVM configuration following the <a href="https://bugs.launchpad.net/nova/+bug/1400315">Kilo</a> enhancement.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><features></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <acpi/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <apic/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <hyperv></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <relaxed state='on'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <vapic state='on'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <spinlocks state='on' retries='8191'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> </hyperv></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> </features></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> ....</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <clock offset='localtime'></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <timer name='pit' tickpolicy='delay'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <timer name='rtc' tickpolicy='catchup'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <timer name='hpet' present='no'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <timer name='hypervclock' present='yes'/></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> </clock></span></div>
</div>
<div>
<br /></div>
<div>
These changes have led to an improvement when running on a loaded hypervisors, especially for Windows 7 and 2008R2 guests. A <a href="https://bugs.launchpad.net/openstack-manuals/+bug/1662991">bug</a> has been opened for the documentation to explain the setting is not Xen only.</div>
<br />
<h2>
Acknowledgements</h2>
<div>
<ul>
<li>Jose Castro Leon performed all of the analysis and testing of the various solutions.</li>
</ul>
</div>
<h2>
References</h2>
<div>
<ul>
<li>Property keys documentation for OpenStack at <a href="http://docs.openstack.org/cli-reference/glance-property-keys.html">http://docs.openstack.org/cli-reference/glance-property-keys.html</a></li>
<li>Additional functionality added to Nova to add hyper-v timer enlightenments for windows guests at <a href="https://bugs.launchpad.net/nova/+bug/1400315">https://bugs.launchpad.net/nova/+bug/1400315</a></li>
<li>Documentation bug report at <a href="https://bugs.launchpad.net/openstack-manuals/+bug/1662991">https://bugs.launchpad.net/openstack-manuals/+bug/1662991</a></li>
<li>Red Hat articles at <a href="https://access.redhat.com/solutions/755943">https://access.redhat.com/solutions/755943</a> and <a href="https://access.redhat.com/node/1471613">https://access.redhat.com/node/1471613</a></li>
</ul>
</div>
<div>
<br /></div>
<br />
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:1;
mso-generic-font-family:roman;
mso-font-format:other;
mso-font-pitch:variable;
mso-font-signature:0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073786111 1 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style>Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com1tag:blogger.com,1999:blog-6032896665180559.post-59143492352973626202017-01-09T06:54:00.000-08:002017-01-09T10:08:01.055-08:00Containers on the CERN cloudWe have recently made the Container-Engine-as-a-Service (Magnum) available in production at CERN as part of the CERN IT department services for the LHC experiments and other CERN communities. This gives the OpenStack cloud users Kubernetes, Mesos and Docker Swarm on demand within the accounting, quota and project permissions structures already implemented for virtual machines.<br />
<br />
We shared the latest news on the service with the CERN technical staff (<a href="https://cernbox.cern.ch/index.php/s/PixyT67aIIaHnva">link</a>). This is the follow up on the tests presented at the OpenStack Barcelona (<a href="https://www.openstack.org/videos/video/toward-10000-containers-on-openstack">link</a>) and covered in the <a href="https://developer.ibm.com/opentech/2016/11/16/how-do-containers-scale-on-openstack/">blog</a> from IBM. The work has been helped by collaborations with Rackspace in the framework of the <a href="http://openlab.cern/about/industry_members/rackspace">CERN openlab</a> and the European Union Horizon 2020 <a href="https://www.indigo-datacloud.eu/">Indigo Datacloud</a> project.<br />
<div>
<ul>
</ul>
<h2>
Performance</h2>
</div>
<div>
At the Barcelona summit, we presented with Rackspace and IBM regarding our additional performance tests after the previous <a href="https://openstack-in-production.blogspot.ch/2016/06/scaling-magnum-and-kubernetes-2-million.html">blog post</a>. We expanded beyond the 2M requests/s to reach around 7M where some network infrastructure issues unrelated to OpenStack limited the scaling further.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-FnyR5mHq6Yg/WFJz3AF9JYI/AAAAAAAATWc/DseQzkvPwocfG2N_J1FVUz2HzjYg1ScLQCLcB/s1600/kub7m.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="618" src="https://4.bp.blogspot.com/-FnyR5mHq6Yg/WFJz3AF9JYI/AAAAAAAATWc/DseQzkvPwocfG2N_J1FVUz2HzjYg1ScLQCLcB/s640/kub7m.png" width="640" /></a></div>
<div>
As we created the clusters, the deployment time increased only slightly with the number of nodes as most of the work is done in parallel. But for 128 node or larger clusters, the increase in time started to scale almost linearly. At the Barcelona summit, the Heat and Magnum teams worked together to develop proposals for how to improve further in future releases, although a 1000 node cluster in 23 minutes is still a good result</div>
<div>
<br /></div>
<div>
<style>
<!--tr
{mso-height-source:auto;}
col
{mso-width-source:auto;}
td
{padding-top:1.0px;
padding-right:1.0px;
padding-left:1.0px;
mso-ignore:padding;
color:windowtext;
font-size:18.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:Arial;
mso-generic-font-family:auto;
mso-font-charset:0;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;}
.oa1
{border:1.0pt solid black;
text-align:center;
vertical-align:top;
padding-bottom:9.6pt;
padding-left:9.6pt;
padding-top:9.6pt;
padding-right:9.6pt;}
.oa2
{border:1.0pt solid black;
background:#D0E0E3;
mso-pattern:auto none;
text-align:center;
vertical-align:top;
padding-bottom:9.6pt;
padding-left:9.6pt;
padding-top:9.6pt;
padding-right:9.6pt;}
-->
</style>
<!--StartFragment-->
<br />
<table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse; mso-yfti-tbllook: 1536; width: 476px;">
<colgroup><col span="3" style="mso-width-source: userset; width: 159pt;" width="159"></col>
</colgroup><tbody>
<tr height="51" style="height: 51.2pt; mso-height-source: userset;">
<td class="oa1" height="51" style="height: 51.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt; font-weight: bold;">Cluster Size (Nodes)</span></div>
</td>
<td class="oa1" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt; font-weight: bold;">Concurrency</span></div>
</td>
<td class="oa1" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt; font-weight: bold;">Deployment Time (min)</span></div>
</td>
</tr>
<tr height="35" style="height: 35.2pt; mso-height-source: userset;">
<td class="oa2" height="35" style="height: 35.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">2</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">50</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">2.5</span></div>
</td>
</tr>
<tr height="35" style="height: 35.2pt; mso-height-source: userset;">
<td class="oa2" height="35" style="height: 35.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">16</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">10</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">4</span></div>
</td>
</tr>
<tr height="35" style="height: 35.2pt; mso-height-source: userset;">
<td class="oa2" height="35" style="height: 35.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">32</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">10</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">4</span></div>
</td>
</tr>
<tr height="35" style="height: 35.2pt; mso-height-source: userset;">
<td class="oa2" height="35" style="height: 35.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">128</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">5</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">5.5</span></div>
</td>
</tr>
<tr height="35" style="height: 35.2pt; mso-height-source: userset;">
<td class="oa2" height="35" style="height: 35.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">512</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">1</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">14</span></div>
</td>
</tr>
<tr height="35" style="height: 35.2pt; mso-height-source: userset;">
<td class="oa2" height="35" style="height: 35.2pt; width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">1000</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">1</span></div>
</td>
<td class="oa2" style="width: 159pt;" width="159"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="font-size: 13pt;">23</span></div>
</td>
</tr>
</tbody></table>
<!--EndFragment--></div>
<br />
<h2>
Storage</h2>
<div>
With the LHC producing nearly 50PB this year, High Energy Physics has some custom storage technologies for specific purposes, EOS for physics data, CVMFS for read-only, highly replicated storage such as applications.<br />
<br />
One of the features of providing a private cloud service to the CERN users is to combine the functionality of open source community software such as OpenStack with the specific needs for high energy physics. For these to work, some careful driver work is needed to ensure appropriate access while ensuring user rights. In particular,<br />
<ul>
<li><a href="https://eos.web.cern.ch/content/about-eos">EOS</a> provides a disk-based storage system providing high-capacity and low-latency access for users at CERN. Typical use cases are where scientists are analysing data from the experiments.</li>
<li><a href="https://cernvm.cern.ch/portal/filesystem">CVMFS</a> is used for a scalable, reliable and low-maintenance for read-only data such as software.</li>
</ul>
<div>
There are also other storage solutions we use at CERN such as</div>
<div>
<ul>
<li><a href="http://clouddocs.web.cern.ch/clouddocs/containers/tutorials/hdfs.html">HDFS</a> for long term archiving of data using Hadoop which uses an HDFS driver within the container. HDFS works in user space, so no particular integration was required to use it from inside (unprivileged) containers</li>
<li>Cinder provides additional disk space using volumes if the basic flavor does not have sufficient. This Cinder integration is offered by upstream Magnum, and work was done in the last OpenStack cycle to improve security by adding support for Keystone trusts.</li>
</ul>
</div>
</div>
<div>
CVMFS was more straightforward as there is no need to authenticate the user. The data is read-only and can be exposed to any container. The access to the file system is provided using a driver (<a href="https://gitlab.cern.ch/cloud-infrastructure/docker-volume-cvmfs/tree/dockerize">link</a>) which has been adapted to run inside a container. This saves having to run additional software inside the VM hosting the container.</div>
<div>
<br /></div>
<div>
EOS requires authentication through mechanisms such as Kerberos to identify the user and thus determine what files they have access to. Here a container is run per user so that there is no risk of credential sharing. The details are in the driver (<a href="https://gitlab.cern.ch/cloud-infrastructure/docker-volume-eos">link</a>).<br />
<br /></div>
<h2>
Service Model</h2>
<div>
One interesting question that came up during the discussions of the container service was how to deliver the service to the end users. There are several scenarios:</div>
<div>
<ol>
<li>The end user launches a container engine with their specifications but they rely on the IT department to maintain the engine availability. This implies that the VMs running the container engine are not accessible to the end user.</li>
<li>The end user launches the engine within a project that they administer. While the IT department maintains the templates and basic functions such as the Fedora Atomic images, the end user is in control of the upgrades and availability.</li>
<li>A variation of option 2., where the nodes running containers are reachable and managed by the end user, but the container engine master nodes are managed by the IT department. This is similar to the current offer from the Google Container Engine and requires some coordination and policies regarding upgrades</li>
</ol>
<div>
Currently, the default Magnum model is for the 2nd option and adding option 3 is something we could do in the near future. As users become more interested in consuming containers, we may investigate the 1st option further<br />
<br /></div>
</div>
<h2>
Applications</h2>
<div>
Many applications at use in CERN are in the process of being reworked for a microservices based architecture. A choice of different container engines is attractive for the software developer. One example of this is the file transfer service which ensures that the network to other high energy physics sites is kept busy but not overloaded with data transfers. The work to containerise this application was described at the recent <a href="http://chep2016.org/">CHEP 2016</a> <a href="https://indico.cern.ch/event/505613/contributions/2227329/attachments/1343962/2035344/Poster-37.pdf">FTS poster</a>.</div>
<div>
<br /></div>
<div>
While deploying containers is an area of great interest for the software community, the key value comes from the physics applications exploiting containers to deliver a new way of working. The <a href="https://indico.cern.ch/event/517393/contributions/2150317/attachments/1269202/1879983/SWAN_SFTmeeting_090516.pdf#search=swan%20openstack">Swan</a> project provides a tool for running <a href="https://root.cern.ch/">ROOT</a>, the High Energy Physics application framework, in a browser with easy access to the storage outlined above. A set of examples can be found at <a href="https://swan.web.cern.ch/notebook-galleries">https://swan.web.cern.ch/notebook-galleries</a>. With the academic paper, the programs used and the data available from the notebook, this allows easy sharing with other physicists during the review process using <a href="https://indico.cern.ch/event/538540/contributions/2187138/attachments/1282513/1906054/IT-cernbox-2016-05-31.pdf">CERNBox</a>, CERN's <a href="https://owncloud.org/">owncloud</a> based file sharing solution.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-rq_LuYL9fdI/WF1EdiyK3eI/AAAAAAAATXI/UHlrNC9p_xocw5Ft-e9LfehlYvdN3brjgCLcB/s1600/swangrab.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="339" src="https://3.bp.blogspot.com/-rq_LuYL9fdI/WF1EdiyK3eI/AAAAAAAATXI/UHlrNC9p_xocw5Ft-e9LfehlYvdN3brjgCLcB/s640/swangrab.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Another application being studied is <a href="http://opendata.cern.ch/?ln=en">http://opendata.cern.ch/?ln=en</a> which allows the general public to run analyses on LHC open data. Typical applications are <a href="https://en.wikipedia.org/wiki/Citizen_science">Citizen Science</a> and outreach for schools.</div>
<div>
<br /></div>
<h2>
Ongoing Work</h2>
<div>
There are a few major items where we are working with the upstream community:</div>
<div>
<ul>
<li>Cluster upgrades will allow us to upgrade the container software. Examples of this would be a new version of Fedora Atomic, Docker or the container engine. With a load balancer, this can be performed without downtime (<a href="https://review.openstack.org/#/c/392193">spec</a>)</li>
<li>Heterogeneous cluster support will allow nodes to have different flavors (cpu vs gpu, different i/o patterns, different AZs for improved failure scenarios). This is done by splitting the cluster nodes into node groups (<a href="https://blueprints.launchpad.net/magnum/+spec/nodegroups">blueprint</a>)</li>
<li>Cluster monitoring to deploy Prometheus and cAdvisor with Grafana dashboards for easy monitoring of a Magnum cluster (<a href="https://blueprints.launchpad.net/magnum/+spec/container-monitoring">blueprint</a>).</li>
</ul>
<div>
<br /></div>
</div>
<h2>
References</h2>
<div>
<ul>
<li>End user documentation for containers on the CERN cloud at <a href="http://clouddocs.web.cern.ch/clouddocs/containers/index.html">http://clouddocs.web.cern.ch/clouddocs/containers/index.html</a></li>
<li>CERN IT department information is at <a href="http://cern.ch/it">http://cern.ch/it</a>.</li>
<li>CERN <a href="http://openlab.cern/about/industry_members/rackspace">openlab</a> Rackspace collaboration on container presentations are listed<a href="http://openlab.cern/about/industry_members/rackspace"> here</a>.</li>
<li>Indigo Datacloud project details are <a href="https://www.indigo-datacloud.eu/">here</a>.</li>
</ul>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com39tag:blogger.com,1999:blog-6032896665180559.post-81097141797716230952016-09-29T09:28:00.003-07:002016-10-05T09:56:59.143-07:00Hyperthreading in the cloudThe cloud at CERN is used for a variety of different purposes from running personal VMs for development/test, bulk throughput computing to analyse the data from the Large Hadron Collider to long running services for the experiments and the organisation.<br />
<div>
<br /></div>
<div>
The configuration of many of the hypervisors is carefully tuned to maximise the compute throughput, i.e. getting as much compute work done in a given time rather than optimising the individual job performance. Many of the workloads are also nearly all embarrassingly parallel, i.e. each unit of compute can be run without needing to communicate with other jobs. A few workloads, such as QCD, need classical High Performance Computing but these are running on dedicated clusters with Infiniband interconnect compared to the typical 1Gbit/s or 10Gbit/s ethernet for the typical hypervisor.<br />
<br class="Apple-interchange-newline" />
CERN has a public procurement procedure which awards tenders to the bid with the lowest price for a given throughput compliant with the specifications. The typical CERN hardware configuration is based on a dual socket configuration and must have at least 2GB/core.</div>
<div>
<br /></div>
<div>
Intel provides a capability for doubling the number of cores on the underlying processor called Simultaneous multithreading or <a href="https://en.wikipedia.org/wiki/Simultaneous_multithreading">SMT</a>. From the machine perspective, this appears as double the number of cores compared to non-SMT configurations. Enabling SMT requires a BIOS parameter change so resources need to be defined in advance and appropriate capacity planning to define the areas of the cloud which are SMT on or off statically.<br />
<br /></div>
<div>
The second benefit of an SMT off configuration is the memory per core doubles. A server with 32 SMT on cores and 64GB of memory with hyper-threading has 2GB per core. A change to 16 cores by dropping SMT leads to 4GB per core which can be useful for some workloads.</div>
<div>
<br /></div>
<div>
Setting the BIOS parameters for a subset of the hypervisors causes multiple difficulties</div>
<div>
<ul>
<li>With older BIOSes, this is a manual operation. New tools are available on the most recent hardware so this is an operation which can be performed with a Linux program and a reboot.</li>
<li>A motherboard replacement requires that the operation is repeated. This can be overlooked as part of the standard repair activities.</li>
<li>Capacity planning requires allocation of appropriate blocks of servers. At CERN, we use OpenStack cells to allow the cloud to scale to our needs with each cells having a unique hardware configuration such as particular processor/memory configuration and thus dedicated cells need to be created for the SMT off machines. When these capacities are exceeded, the other unused cloud resources cannot be trivially used but further administration reconfiguration is required.</li>
</ul>
<div>
The reference benchmark for High Energy Physics is <a href="https://w3.hepix.org/benchmarks/doku.php">HEPSpec06</a>, a subset of the Spec benchmarks which match the typical instruction workload. Using this, run in parallel on each of the cores in a machine, the throughput provided by a given configuration can be measured.<br />
<br /></div>
<table align="center" border="1">
<tbody>
<tr>
<th>SMT</th>
<th>VM configuration</th>
<th>Throughput HS06</th>
</tr>
<tr><td>On</td><td>2 VMs each 16 cores</td><td align="right">351</td></tr>
<tr><td>On</td><td>4 VMs each 8 cores</td><td align="right">355</td></tr>
<tr><td>Off</td><td>1 VM of 16 cores</td><td align="right">284.5</td></tr>
</tbody></table>
<br />
Thus, the total throughput of the server with SMT off is significantly less (284.5 compared to 351) but the individual core performance is higher (284.5/16=17.8 compared to 351/32=11). Where an experiment workflow is serialised for some of the steps, this higher single core performance was a significant gain, but at an operational cost.<br />
<br />
To find a cheaper approach, the recent additions of NUMA flavors in OpenStack was used. The hypervisors were configured with SMT on but a flavor was created to only use half of the cores on the server with 4GB/core so that the hypervisors were under committed on cores but committed by memory to avoid another VM being allocated to the unused cores. In our configuration, this was done by adding <span style="font-family: "courier new" , "courier" , monospace;">numa_nodes=2</span> to the flavor and the NUMA aware scheduler does the appropriate allocation.<br />
<br />
This configuration was benchmarked and compared with the SMT On/Off.<br />
<br />
<table align="center" border="1">
<tbody>
<tr>
<th>SMT</th>
<th>VM configuration</th>
<th>Throughput HS06</th>
</tr>
<tr><td>On</td><td>2 VMs each 16 cores</td><td align="right">351</td></tr>
<tr><td>Off</td><td>1 VM of 16 cores</td><td align="right">284.5</td></tr>
<tr><td>On</td><td>1 VM of 16 cores with numa_nodes=2</td><td align="right">283</td></tr>
</tbody></table>
<br />
The new flavor shows similar characteristics to the SMT Off configuration without requiring the BIOS setting change and can therefore be deployed without needing the configuration of dedicated cells with a particular hardware configuration. The Linux and OpenStack schedulers appear to be allocating the appropriate distribution of cores across the processors.<br />
<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com6tag:blogger.com,1999:blog-6032896665180559.post-8093378351165902142016-09-22T06:46:00.001-07:002016-09-24T09:25:10.309-07:00Our Cloud in LibertyWe have previously posted experiences with upgrades of OpenStack such as<br />
<ul>
<li>Kilo - <a href="https://openstack-in-production.blogspot.fr/2015/11/our-cloud-in-kilo.html">https://openstack-in-production.blogspot.fr/2015/11/our-cloud-in-kilo.html</a></li>
<li>Juno - <a href="http://openstack-in-production.blogspot.fr/2015/05/our-cloud-in-juno.html">http://openstack-in-production.blogspot.fr/2015/05/our-cloud-in-juno.html</a></li>
<li>Icehouse - <a href="http://openstack-in-production.blogspot.fr/2014/11/our-cloud-in-icehouse.html">http://openstack-in-production.blogspot.fr/2014/11/our-cloud-in-icehouse.html</a></li>
<li>Havana - <a href="http://openstack-in-production.blogspot.fr/2014/02/our-cloud-in-havana.html">http://openstack-in-production.blogspot.fr/2014/02/our-cloud-in-havana.html</a></li>
</ul>
<div>
The upgrade to Liberty for the CERN cloud was completed at the end of August. Working with the upstream OpenStack, Puppet and RDO communities, this went pretty smoothly without any issues so there is no significant advice to report. We followed the same approach as the past, gradually upgrading component by component. With the LHC reaching it's highest data rates so far this year (over 10PB recorded to tape during June), the upgrades needed to be done without disturbing the running VMs.</div>
<div>
<br /></div>
<div>
Some hypervisors are still on Scientific Linux CERN 6 (Kilo) using Python 2.7 in a software collection but the backwards compatibility has allows the rest of the cloud to migrate while we complete the migration of 5000 VMs from old hardware and SLC6 to new hardware on CentOS 7 in the next few months.<br />
<br />
After the migration, we did encounter a few problems:<br />
<ul>
<li>Live migration from Kilo to Liberty gave an error (<a href="https://bugs.launchpad.net/nova/+bug/1576048">https://bugs.launchpad.net/nova/+bug/1576048</a>)</li>
<li>Cannot delete machines with NUMA topology created before the upgrade (<a href="https://bugs.launchpad.net/nova/+bug/1596119">https://bugs.launchpad.net/nova/+bug/1596119</a>)</li>
<li>With the move to Nova API 2.1, a change in policy files was needed for our 'operator' role (details are at <a href="https://openstack-in-production.blogspot.ch/2015/02/delegation-of-roles.html">https://openstack-in-production.blogspot.ch/2015/02/delegation-of-roles.html</a>). Some additional actions needed to be defined for the operator such as <span style="font-family: "courier new" , "courier" , monospace;">os_compute_api:servers:detail</span>.</li>
</ul>
</div>
<div>
The first two cases are being backported to Liberty so others who have not upgraded may not see these.<br />
<br />
We've already started the Mitaka upgrade with Barbican, Magnum and Heat ahead of the other components as we enable a CERN production container service. The other components will follow in the next six months including the <a href="https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15865/neutron-at-cern-moving-thousands-of-production-nodes-from-nova-network">experiences of the migration to Neutron</a> which we'll share at the OpenStack summit in October.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com9tag:blogger.com,1999:blog-6032896665180559.post-28939441634022427072016-06-17T06:50:00.000-07:002016-07-06T23:32:37.746-07:00Scaling Magnum and Kubernetes: 2 million requests per second<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
Two months ago, we described in <a href="https://openstack-in-production.blogspot.ch/2016/04/containers-and-cern-cloud.html">this blog post</a> how we deployed OpenStack Magnum in the CERN cloud. It is available as a pre-production service and we're steadily moving towards full production mode as a standard part of the CERN IT service offerings to give Containers-as-a-Service.<br />
<br />
As part of this effort, we've started testing the upgrade procedures, the latest being to the final Mitaka release. If you're here to see some fancy load tests, keep reading below, but some interesting details on the upgrade:<br />
<ul style="text-align: left;">
<li>We build our own RPMs to include a few patches from post-Mitaka upstream (the most important being the trustee user to support lifecycle operations on the bays) and some CERN customizations (removal of neutron LBaaS and floating ips which we don't yet have, adding the CERN Certificate Authority, ...). <a href="https://gitlab.cern.ch/cloud-infrastructure/openstack-magnum/tree/cern-mitaka">Check here</a> for the patches and build procedure</li>
<li>We build our Fedora Atomic 23 image to get more recent versions of docker and kubernetes (1.10 and 1.2 respectively), plus support for an internal distributed filesystem called <a href="https://cvmfs.readthedocs.io/en/latest/">CVMFS</a>. We do use the upstream disk-imagebuilder procedure with a few additional elements <a href="https://gitlab.cern.ch/cloud-infrastructure/cci-elements/tree/master">available here</a></li>
</ul>
<div>
While discussing how we could further test the service, we thought of this <a href="http://blog.kubernetes.io/2015/11/one-million-requests-per-second-dependable-and-dynamic-distributed-systems-at-scale.html">kubernetes blog post</a>, achieving 1 million requests per second against a service running on a kubernetes cluster. We thought we could probably do the same. Requirements included:</div>
<div>
<ul style="text-align: left;">
<li>kubernetes 1.2, which our recent upgrade offered</li>
<li>available resources to deploy the cluster, and luckily we were installing a new batch of a few hundred physical nodes which could be used for a day or two</li>
</ul>
<div>
So along with the upgrade, Bertrand and Mathieu got to work to test this setup and we quickly got it up and running.</div>
<div>
<br /></div>
<div>
Quick summary of the setup:<br />
<ul style="text-align: left;">
<li>1 kubernetes bay</li>
<li>1 master node, 16 cores (not really needed but why not)</li>
<li>200 minions, 4 cores each</li>
</ul>
<div>
In total there are 800 cores, which matches the cluster used in the original test. How did our test go?</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-S-MODLZR5Rc/V2P_I-JxIEI/AAAAAAAAkdY/7_JgqM9Tie8DYNeROjYYJJNEjV_ldJW5QCLcB/s1600/1million.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="303" src="https://3.bp.blogspot.com/-S-MODLZR5Rc/V2P_I-JxIEI/AAAAAAAAkdY/7_JgqM9Tie8DYNeROjYYJJNEjV_ldJW5QCLcB/s320/1million.gif" width="320" /></a></div>
<br /></div>
</div>
</div>
<div>
<br />
We ended up trying a bit more and doubled the number to 2 million requests per second :)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-9nw1R_AdUcU/V2P_L-sqOgI/AAAAAAAAkdg/nX8hJ46tpC456dJlW687YH_bvt74L6kAgCKgB/s1600/2million.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="302" src="https://2.bp.blogspot.com/-9nw1R_AdUcU/V2P_L-sqOgI/AAAAAAAAkdg/nX8hJ46tpC456dJlW687YH_bvt74L6kAgCKgB/s320/2million.gif" width="320" /></a></div>
<br />
<br />
We learned a few things on the way:</div>
<div>
<ul style="text-align: left;">
<li>set Heat's <i>max_resources_per_stack</i> to something big. Magnum stacks create a lot of these, and with bays of hundreds of nodes the value gets high enough that unlimited (-1) is tempting and we have it like that now. It leaves the option for people to deploy a stack with so many resources that Heat could break, so we'll investigate what the best value is</li>
<li>while creating and deleting many large bays, Heat shows errors like 'TimeoutError: QueuePool limit of size ... overflow ... reached' which we've seen in the past for other OpenStack services. We'll contribute the patch to fix it upstream if not there yet</li>
<li>latency values get high even before the 1 million barrier, we'll check further the demo code and our setup (using local disk, in this case SSDs instead of the default volume attachment in Magnum should help)</li>
<li>Heat timeout and retrial configuration values need to be tuned to manage very large stacks. We're still not sure what are the best values, but will update the post once we have them</li>
<li>Magnum shows 'Too many files opened' errors, we also have a fix to contribute for this one</li>
<li>Nova, Cinder (bay nodes use a volume), Keystone and all other OpenStack services scaled beautifully, our cloud usually has a rate of ~150 VMs created and deleted per hour, here's the plot for the test period, we eventually tried bays up to 1000 nodes</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-ZcEyraNZTEA/V2OyrbUc-UI/AAAAAAAAkcw/iQ3kREm0FCoOvzjeA1nCkoGH_OP1eWXEwCLcB/s1600/vmstest.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="143" src="https://2.bp.blogspot.com/-ZcEyraNZTEA/V2OyrbUc-UI/AAAAAAAAkcw/iQ3kREm0FCoOvzjeA1nCkoGH_OP1eWXEwCLcB/s400/vmstest.png" width="400" /></a></div>
<br />
<br />
<div>
And what's next? </div>
<div>
<ul style="text-align: left;">
<li><b>Larger bays: </b>at the end of these tests we deployed a few bigger bays with 300, 500 and 1000 nodes. And in just a couple weeks there will be a new batch of physical nodes arriving, so we plan to upgrade Heat to Mitaka and build on the recent upstream work (by Spyros together with Ton and Winnie from IBM) adding Magnum scenarios to Rally to run additional scale tests and see where it breaks</li>
<li><b>Bay lifecycle: </b>we stopped at launching a large number of requests in a bay, next we would like to perform bay operations (update of number of nodes, node replacement) and see which issues (if any) we find in Magnum</li>
<li><b>New features:</b> lots of upstream work going on, so we'll do regular Magnum upgrades (cinder support, improved bay monitoring, support for some additional internal systems at CERN)</li>
</ul>
<div>
And there's also Swarm and Mesos, we plan on testing those soon as well. And kubernetes <a href="http://blog.kubernetes.io/2016/03/1000-nodes-and-beyond-updates-to-Kubernetes-performance-and-scalability-in-12.html">updated their test</a>, so stay tuned...<br />
<br /></div>
</div>
</div>
<h4 style="text-align: left;">
Acknowledgements</h4>
<div>
<ul style="text-align: left;">
<li>Bertrand Noel, Mathieu Velten and Spyros Trigazis from CERN IT, for the work upstream and integrating Magnum at CERN, and on getting these demos running</li>
<li><a href="https://www.rackspace.co.uk/">Rackspace</a> for their support within the <a href="https://openlab.web.cern.ch/">CERN Openlab</a> on running containers at scale</li>
<li><a href="https://www.indigo-datacloud.eu/">Indigo Datacloud</a> building a platform as a service for e-science in Europe</li>
<li>Kubernetes for an awesome tool and the nice demo</li>
<li>All in the CERN OpenStack Cloud team, for a great service (especially Davide Michelino and Belmiro Moreira for all the work integrating Neutron at CERN)</li>
<li>The upstream Magnum team, for building what is now looking like a great service, and we look forward for what's coming next (bay drivers, bare metal support, and much more)</li>
<li>Tim, Arne and Jan for letting us use the new hardware for a few days<div class="separator" style="clear: both; text-align: center;">
<br /></div>
</li>
</ul>
</div>
</div>
Ricardo Rochahttp://www.blogger.com/profile/12954165425097243113noreply@blogger.com33tag:blogger.com,1999:blog-6032896665180559.post-91084619860935042932016-04-30T06:15:00.001-07:002016-05-07T03:20:23.652-07:00Resource management at CERN<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br />
<br />
As part of the recent OpenStack summit in Austin, the <a href="https://wiki.openstack.org/wiki/Scientific_working_group">Scientific Working</a> group was established looking into how scientific organisations can best make use of OpenStack clouds.<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559"></a>During our discussions with more than 70 people (<a href="https://etherpad.openstack.org/p/scientific-wg-austin-summit-agenda">etherpad</a>), we concluded on 4 top areas to look at and started to analyse the approaches and common needs. The areas were<br />
<div>
<ol>
<li>Parallel file system support in Manila. There are a number of file systems supported by Manila but many High Performance Computing sites (HPC) use <a href="http://lustre.org/">Lustre</a> which is focussed on the needs of the HPC user community.</li>
<li>Bare metal management looking at how to deploy bare metal for the maximum performance within the OpenStack frameworks for identity, quota and networking. This team will work on understanding additional needs with the <a href="https://wiki.openstack.org/wiki/Ironic">OpenStack Ironic</a> project.</li>
<li>Accounting covering the wide range of needs to track usage of resources and showback/chargeback to the appropriate user communities.</li>
<li>Stories is addressing how we collect requirements from the scientific use cases and work with the OpenStack community teams, such as the Product working group, to include these into the development roadmaps along with defining reference architectures on how to cover common use cases such as high performance or high throughput computing clouds in the scientific domain.</li>
</ol>
Most of the applications run at CERN are high throughput, embarrassingly parallel applications. Simulation and analysis of the collisions such as in the LHC can be farmed to different compute resources with each event being handled independently and no need for fast interconnects. While the working group will cover all the areas (and some outside this list), our focus is on accounting (3).<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>Given limited time available, it was not possible for each of the interested members of the accounting team to explain their environment. This blog is intended to provide the details of the CERN cloud usage, the approach to resource management and some areas where OpenStack could provide additional function to improve the way we manage the accounting process. Within the Scientific Working group, these stories will be refined and reviewed to produce specifications and identify the potential communities who could start on the development.</div>
<div>
<br /></div>
<h3>
CERN Pledges</h3>
The CERN cloud provides computing resources for the Large Hadron Collider and other experiments. The cloud is currently around 160,000 cores in total spread across two data centres in <a href="https://meter.cern.ch/public/_plugin/kibana/#/dashboard/elasticsearch/Overview:%20Data%20Centre">Geneva and Budapest</a>. Resources are managed world wide with the World Wide Computing LHC Grid which executes over 2 million jobs per day. Compute resources in the <a href="http://wlcg-public.web.cern.ch/">WLCG</a> are allocated via a pledge model. Rather than direct funding from the experiments or WLCG, the sites, supported by their government agencies, commit to provide compute capacity and storage for a period of time as a pledge and these are recorded in the <a href="https://wlcg-rebus.cern.ch/apps/pledges/resources/">REBUS</a> system. These are then made available using a variety of middleware technologies.<br />
<br />
Given the allocation of resources across 100s of sites, the experiments then select the appropriate models to place their workloads at each site according to compute/storage/networking capabilities. Some sites will be suitable for simulation of collisions (high CPU, low storage and network). Others would provide archival storage and significant storage IOPS for more data intensive applications. For storage, the pledges are made in capacity on disk and tape. The compute resource capacity is pledges in Kilo-<a href="https://w3.hepix.org/benchmarks/doku.php">HepSpec06</a> units, abbreviated to kHS06 (based on a subset of the Spec 2006 benchmark) that allows faster processors to be given a higher weight in the pledge compared to slower ones (as High Energy Physics computing is an embarrassingly parallel high throughput computing problem).<br />
<br />
The pledges are reviewed on a regular basis to check the requests are consistent with the experiments’ computing models, the allocated resources are being used efficiently and the pledges are compatible with the requests.<br />
<br />
Within the WLCG, CERN provides the <a href="http://home.cern/about/computing/grid-system-tiers">Tier-0</a> resources for the safe keeping of the raw data and performs the first pass at reconstructing the raw data into meaningful information. The Tier-0 distributes the raw data and the reconstructed output to Tier 1s, and reprocesses data when the LHC is not running.<br />
<div>
<div class="MsoNormal">
<span style="font-family: "times new roman";"><br /></span></div>
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Procurement Process<o:p></o:p></span></b></div>
<div class="MsoNormal">
<br />
The purchases for the Tier-0 pledge for compute is translated into a formal <a href="http://procurement.web.cern.ch/">procurement process</a>. Given the annual orders exceed 750 KCHF, the process requires a formal procedure:<br />
<ul>
<li>A market survey to determine which companies in the <a href="http://home.cern/about/member-states">CERN member states</a> could reply to requests in general areas such as compute servers or disk storage. Typical criteria would be the size of the company, the level of certification with component vendors and offering products in the relevant area (such as industry standard servers) </li>
<li>A tender which specifies the technical specifications and quantity for which an offer is required. These are adjudicated on the lowest cost compliant with specifications criteria. Cost in this case is defined as the cost of the material over 3 years including warranty, power, rack and network infrastructure needed. The quantity is specified in terms of kHS06 with 2GB/core and 20GB storage/core which means that the suppliers are free to try different combinations of top bin processors which may be a little more expensive or lower performing ones which would then require more total memory and storage. Equally, the choice of motherboard components has significant flexibility within the required features such as 19” rack compatible and enterprise quality drives. The typical winning configurations recently have been white box manufacturers.</li>
<li>Following testing of the proposed systems to ensure compliance, an order is placed with several suppliers, the machines manufactured, delivered, racked-up and burnt it using a set of high load stress tests to identify issues such as cooling or firmware problems.</li>
</ul>
</div>
<div class="MsoNormal">
Typical volumes are around 2,000 servers a year in one or two rounds of procurement. The process from start to delivered capacity takes around 280 days so bulk purchases are needed followed by allocation to users rather than ordering on request. If there are issues found, this process can take significantly longer.<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse; mso-yfti-tbllook: 1056; width: 648px;">
<colgroup><col style="mso-width-source: userset; width: 237pt;" width="237"></col>
<col style="mso-width-source: userset; width: 236pt;" width="236"></col>
<col style="mso-width-source: userset; width: 175pt;" width="175"></col>
</colgroup><tbody>
<tr height="26" style="height: 26.27pt; mso-height-source: userset;">
<td class="oa1" height="26" style="height: 26.27pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #0055a0; font-family: "arial"; font-size: 14.0pt; font-weight: bold;">Step</span></div>
</td>
<td class="oa1" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #0055a0; font-family: "arial"; font-size: 14.0pt; font-weight: bold;">Time (Days)</span></div>
</td>
<td class="oa1" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #0055a0; font-family: "arial"; font-size: 14.0pt; font-weight: bold;">Elapsed (Days)</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa2" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">User expresses requirement</span></div>
</td>
<td class="oa3" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
</div>
</td>
<td class="oa3" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">0</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa4" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Market</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt; vertical-align: baseline;"> Survey prepared</span></div>
</td>
<td class="oa5" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">15</span></div>
</td>
<td class="oa5" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">15</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa6" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Market Survey for possible vendors</span></div>
</td>
<td class="oa7" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">30</span></div>
</td>
<td class="oa7" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">45</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa4" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Specifications prepared</span></div>
</td>
<td class="oa5" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">15</span></div>
</td>
<td class="oa5" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">60</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa6" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Vendor responses</span></div>
</td>
<td class="oa7" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt; vertical-align: baseline;">30</span></div>
</td>
<td class="oa7" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">90</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa4" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Test systems evaluated</span></div>
</td>
<td class="oa5" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">30</span></div>
</td>
<td class="oa5" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">120</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa6" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Offers adjudicated</span></div>
</td>
<td class="oa7" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">10</span></div>
</td>
<td class="oa7" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">130</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa4" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Finance</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt; vertical-align: baseline;"> committee</span></div>
</td>
<td class="oa5" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">30</span></div>
</td>
<td class="oa5" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">160</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa6" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">Hardware</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt; vertical-align: baseline;"> delivered</span></div>
</td>
<td class="oa7" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">90</span></div>
</td>
<td class="oa7" style="width: 175pt;" width="175"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14.0pt;">250</span></div>
</td>
</tr>
<tr height="30" style="height: 29.59pt; mso-height-source: userset;">
<td class="oa4" height="30" style="height: 29.59pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt;">Burn</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt; vertical-align: baseline;"> in and acceptance</span></div>
</td>
<td class="oa5" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: right; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt;">30</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt; vertical-align: baseline;"> days typical with 38</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt;">0 worst</span><span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt; vertical-align: baseline;">
case</span></div>
</td>
<td class="oa5" style="width: 175pt;" width="175"><div style="direction: ltr; margin: 0pt 0in; text-align: right; text-indent: 0in; unicode-bidi: embed; word-break: normal;">
<span style="color: #5f5f5f; font-family: "arial"; font-size: 14pt;">280</span></div>
</td>
</tr>
<tr height="22" style="height: 21.9pt; mso-height-source: userset;">
<td class="oa6" height="22" style="height: 21.9pt; width: 237pt;" width="237"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
<span style="color: #0055a0; font-family: "arial"; font-size: 14.0pt; font-weight: bold;">Total</span></div>
</td>
<td class="oa6" style="width: 236pt;" width="236"><div style="direction: ltr; margin-bottom: 0pt; margin-left: 0in; margin-top: 0pt; unicode-bidi: embed; word-break: normal;">
</div>
</td>
<td class="oa7" style="width: 175pt;" width="175"><div style="direction: ltr; margin: 0pt 0in; text-align: right; text-indent: 0in; unicode-bidi: embed; word-break: normal;">
<span style="color: #0055a0; font-family: "arial"; font-size: 14.0pt; font-weight: bold;">280+ Days</span></div>
</td>
</tr>
</tbody></table>
<!--EndFragment--><br />
<span style="font-family: "times new roman";">Given the time the process takes, there are only one to two procurement processes run per year. This means that a continuous delivery model cannot be used and therefore there is a need for capacity planning on an annual basis and to find approaches to use the resources before they are allocated out to their final purpose.</span><br />
<span style="font-family: "times new roman";"><br /></span>
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Physical Infrastructure</span></b></div>
<div class="MsoNormal">
<span style="font-family: "times new roman";"><br /></span>
<span style="font-family: "times new roman";">CERN manages two
data centres in Meyrin, Geneva and Wigner, Budapest. The full data is available at the </span><a href="https://meter.cern.ch/public/_plugin/kibana/#/dashboard/elasticsearch/Overview:%20Data%20Centre" style="font-family: 'times new roman';">CERN data centre overview</a><span style="font-family: "times new roman";"> page. When hardware is procured, the final destination is defined as part of the order according to rack space, cooling and electrical availability. </span><br />
<span style="font-family: "times new roman";"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-39yEUgC5yTA/VyOXyu2VHlI/AAAAAAAASs8/BmwD-YMr24898GEfjmoIQJwm7senOb4xACLcB/s1600/ccbynumbers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="224" src="https://2.bp.blogspot.com/-39yEUgC5yTA/VyOXyu2VHlI/AAAAAAAASs8/BmwD-YMr24898GEfjmoIQJwm7senOb4xACLcB/s640/ccbynumbers.png" width="640" /></a></div>
<br />
<br />
While the installations in Budapest are new installations, some of the Geneva installations involve replacing old hardware. We typically retire hardware between 4 and 5 years old when the CPU power/watt is significantly better with new purchases and the hardware repair costs for new equipment are more predictable and sustainable.<br />
<br />
Within the Geneva centre, there are two significant areas, physics and redundant power. Physics power has a single power source which is expected to fail in the event of an electricity cut lasting beyond the few minutes supported by the battery units. The redundant power area is backed by diesels. The Wigner centre is entirely redundant.<br />
<br /></div>
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Lifecycle</span></b></div>
<br />
With an annual procurement cycle with 2-3 vendors per cycle, each one with their own optimisations to arrive at the lowest cost for the specifications, the hardware is highly heterogeneous. This has a significant benefit when there are issues, such as disk firmware or BMC controllers, that lead to delays in one of the deliveries being accepted, so the remaining hardware can be made available to experiments.<br />
<br />
However, we run the machines for the 3 year warranty and then some additional years on minimal repairs (i.e. simple parts are replaced with components from servers of the same series), we have around 15-20 different hardware configurations for compute servers active in the centre at any time. There are variations in the specifications (as technologies such as SSDs and 10Gb Ethernet became commodity, the new tenders needed these) and those between vendor responses for the same specifications (e.g. slower memory or different processor models). <br />
<br />
These combinations do mean that offering standard flavors for each hardware complication would be very confusing for the users, given that there is no easy way for a user to know if resources are available in a particular flavor except to try to create a VM with that flavor. <br />
<br />
Given new hardware deliveries and limited space, there are equivalent retirement campaigns. The aim is to replace the older hardware by more efficient newer boxes that can deliver more HS06 within the same power/cooling envelope. The process to empty machines depends on the workloads running on the servers. Batch workloads generally finish within a couple of weeks so setting the servers to no longer accept new work just before the retirements is sufficient. For servers and personal build/test machines, we aim to migrate the workloads to capacity on new servers. This operation is increasingly being performed using live migration and MPLS to extend the broadcast domains for networks to the new capacity.<br />
<br />
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Projects and Quota</span></b><br />
<br />
All new users are allocated a project, “Personal XXXX” where XXXX is their CERN account when they subscribe to the CERN cloud service through the CERN resource portal. The CERN resource portal is the entry point to subscribe to the many services available from the central IT department and for users to list their currently active subscriptions and allocations. The personal projects have a minimal quota for a few cores and GBs of block storage so that users can easily follow the tutorial steps on using the cloud and create simple VMs for their own needs. The default image set is available on personal projects along with the standard ‘m’ flavors which are similar to the ones on AWS.<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559"></a>Shared projects can also be requested for activities which are related to an experiment or department. For these resources, a list of people can be defined as administrators (through CERN’s group management system e-groups) and a quota for cores, memory and disk space requested. Additional flavors can also be asked for according to particular needs such as the CERNVM flavors with a small system disk and a large ephemeral one.<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559"></a>The project requests go through a manual approval process, being reviewed by the IT resource management to check the request against the pledge and the available resources. An adjustment of the share of the central batch farm is also made so that the sum of resources for an experiment continues to be within the pledge.<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559"></a>Once the resource request has been manually approved, the ticket is then passed to the cloud team for execution. <a href="https://indico.cern.ch/event/466991/contributions/1143619/attachments/1261496/1864739/rundeck-hepix_cern.pptx">Rundeck</a> provides us with a simple tool for performing high privilege operations with good logging and recovery. This is used in many of our <a href="https://indico.cern.ch/event/466991/contributions/1143619/attachments/1261496/1864739/rundeck-hepix_cern.pptx">workflows</a> such as hardware repair. The Rundeck procedure reads the quota settings from the ticket and executes the appropriate project creation and role allocation requests to Keystone, Nova and Cinder.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Need #1 : CPU performance based allocation and scheduling</span></b><br />
<br />
As with all requests, there is a mixture of requirements and implementation. The needs are stated according to our current understanding. There may be alternative approaches or compromises which would address these needs in common with other user requirements. One of the aims of the Scientific Working group is to expose these ideas to other similar users, adapt them to meet the general community and work with the product working group, user committee and developers on an approach. Thus, the subsequent areas where CERN would be interested in improvements in the underlying software techologies.<br />
<br />
The resource manager for the cloud in a high throughput/performance computing environment allocates resources based on performance rather than pure core count.<br />
<br />
A user wants to request a processor of a particular minimum performance and is willing to have a larger reduction in his remaining quota.<br />
<br />
A user wants to make a request for resources and then iterate according to how much performance related quota they have left to fill the available quota, e.g. give me a VM with less than a certain performance rating.<br />
<br />
A resource manager would like to encourage the use of older resources which are often idle. <br />
<br />
A quotas for slower and faster cores is currently the same (thus users create a VM, delete it if it is one of the slower type) so there is no incentive to use the slower cores.<br />
<br />
As a resource manager preparing an accounting report, the faster cores should have a higher weight against the pledge to ensure continued treatment of slower cores.<br />
<br />
The proposal is therefore to have an additional, optional quota on CPU units so that resource managers can allocate out total throughput rather than per core capacities. <br />
<br />
The alternative approach of defining a number of flavors for each of the hardware types and quota. However, there are a number of drawbacks with this approach:</div>
</div>
<div>
<ul>
<li>The number of flavors to define would be significant (in CERN’s case, around 15-20 different hardware configurations multiplied by 4-5 sizes for small, medium, large, xlarge, ...)</li>
<li>The user experience impact would be significant as the user would have to iterate over the available flavors to find free capacity. For example, trying out m4.large first and finding the capacity was all used, then trying m3.large etc.</li>
</ul>
There is, as far as I know, no per-flavor quota. The specs have been discussed in some operator feedback sessions but the specification did not reach consensus.<br />
<br />
<div>
Extendible resource tracking seems to be approaching the direction with ‘compute units’ (such as <a href="https://wiki.openstack.org/wiki/HeterogeneousInstanceTypes">here</a>) as defined in the specification. Many parts of this are not yet implemented so it is not easy to see if it addresses the requirements.</div>
<br />
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Need #2 : Nested Quotas<o:p></o:p></span></b></div>
<br />
As an experiment resource co-ordinator, it should be possible to re-allocate resources according to the priorities of the experiment without needing action by the administrators. Thus, moving quotas between projects which are managed by the experiment resource co-ordinators within the pledge allocated by the WLCG.<br />
<br />
Nested keystone projects have been in the production release since Kilo. This gives the possibility for role definitions within the nested project structure.</div>
<div>
<br /></div>
<div>
The implementation of the nested quota function has been discussed within various summits for the past 3 years. The first implementation proposal, <a href="https://wiki.openstack.org/wiki/Boson">Boson</a>, was for a dedicated service for quota management. However, there were concerns raised by the PTLs on the impacts for performance and maintainability of this approach. The alternative of enhancing the quotas in each of the projects has been followed (such as <a href="https://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/nested-quota-driver-api.html">Nova</a>). These implementations though have not advanced due to other concerns with quota management which are leading towards a common library, <a href="https://review.openstack.org/#/c/284454/">delimiter</a>, which is being discussed for Newton.</div>
<div>
<b><span style="font-family: "times new roman"; font-size: 13.5pt;"><br /></span></b>
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Need #3 : Spot Market</span></b><br />
<br />
As a cloud provider, uncommitted resources should be made available at a lower cost but at a lower service level, such as pre-emption and termination at short notice. This mirrors the AWS spot market or the Google Pre-emptible instances. The benefits would be higher utilization of the resources and ability to provide elastic capacity for reserved instances by reducing the spot resources.<br />
<br />
A draft <a href="https://blueprints.launchpad.net/nova/+spec/spot-instances">specification</a> for this functionality has been submitted along with the <a href="https://review.openstack.org/#/c/104883/">proposal </a>which is currently being reviewed. An initial implementation of this functionality (called OpenStack Preemptible Instances Extension, or opie) will be made available soon on <a href="https://github.com/indigo-dc/opie">github</a> following the work on <a href="https://www.indigo-datacloud.eu/">Indigo Datacloud</a> by <a href="https://grid.ifca.es/">IFCA</a>. A demo video is available on <a href="https://www.youtube.com/watch?v=c0f0fag3RGE">YouTube</a>.</div>
<div>
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br />
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Need #4 : Reducing quota below
utilization<o:p></o:p></span></b></div>
<br />
As an experiment resource co-ordinator, quotas are under regular adjustment to meet the chosen priorities. Where a project has a lower priority but high current utilization, further resource usage should be blocked but existing resources not deleted since the user may still need to complete the processing on those VMs. The resource co-ordinator can then contact the user to encourage the appropriate resources to be deleted.<br />
<br />
To achieve this function, one approach would be to allow the quota to be set below the current utilization in order to give the project administrator the time to identify the resources which would be best to be deleted in view of the reduced capacity.</div>
<div>
<br />
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Need #5 : Components without quota<o:p></o:p></span></b></div>
<br />
As a cloud provider, some inventive users are storing significant quantities of data in the Glance image service. There is only a maximum size limit with no accumulated capacity leaves this service open to non-planned storage.<br />
<br />
It is proposed to add quota functionality inside Glance for</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a></div>
<div>
<ul>
<li>The total capacity of images stored in Glance</li>
<li>The total capacity of snapshots stored in Glance</li>
</ul>
<a href="https://www.blogger.com/blogger.g?blogID=6032896665180559"></a>The number of images and snapshots would be an lower priority enhancement request since the service risk comes from the total capacity although the numbers could potentially also be abused.<br />
<br />
Given that this is new functionality, it could also be a candidate for the first usage of the new <a href="http://www.slideshare.net/vilobh/delimiter-openstack-cross-project-quota-library-proposal">delimiter</a> library.</div>
<br />
<div class="MsoNormal">
<b><span style="font-family: "times new roman"; font-size: 13.5pt;">Need #6 : VM Expiration<o:p></o:p></span></b></div>
<br />
As a private cloud provider, some VMs should be time limited to ensure that they are still required by their end users and automatically expire if there is no confirmation. Personal projects are often in this category as users will launch test instances but fail to delete them on completion of the test. These cases the users should be required to confirm that they will need the resources and that these are within their current allocated quota (Need #4).<br />
<h3>
Acknowledgements</h3>
There are too many people who have been involved in the resource management activities to list here. The teams contributing to the description above are:<br />
<ul>
<li>CERN IT teams supporting cloud, batch, accounting and quota</li>
<li>BARC, Mumbai for collaborating around the implementation of nested quota</li>
<li>Indigo Datacloud team for work on the spot market in OpenStack</li>
<li>Other labs in the WLCG and the Scientific Working Group</li>
</ul>
<div>
<br /></div>
<br />
<br />
<br />
<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br />
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com286tag:blogger.com,1999:blog-6032896665180559.post-23458179005716699482016-04-21T10:29:00.001-07:002016-04-21T10:29:05.869-07:00Containers and the CERN cloud<div dir="ltr" style="text-align: left;" trbidi="on">
In recent years, different groups at CERN started looking at using containers for different purposes, covering infrastructure services but also end user applications. These efforts have been mostly done independently, resulting in a lot of repeated work especially for the parts which are CERN specific: integration with the identity service, networking and storage systems. In many cases, the projects could not complete before reaching a usable state, as some of these tasks require significant expertise and time to be done right. Alternatively, they found different solutions to the same problem which led to further complexity for the supporting infrastructure services. However, the use cases were real, and a lot of knowledge had been built on the available tools and their capabilities.<br />
<br />
Based on this, we started a project with the following goals:<br />
<ul style="text-align: left;">
<li>integrate containers into the CERN OpenStack cloud, building on top of already available tools such as resource lifecycle, quotas, identity and authorization</li>
<li>stay container orchestration agnostic, allowing users to select any of the most common solutions (Docker Swarm, Kubernetes, Mesos)</li>
<li>allow fast cluster deployment and rebuild</li>
</ul>
We had done a prototype using the Nova LXC driver in the past but the long term support was not clear and we wanted access to the native container functions using the standard tools.<br />
<br />
Looking for other possibilities, OpenStack Magnum seemed to be offering a lot of what we needed, and we decided to try it out. At around the same time we were also heading to the OpenStack Tokyo summit, which was a great opportunity to follow the Magnum sessions and learn more of what it provides.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-xbOT-PKvXiA/VxaoJ1ZMTXI/AAAAAAAAjoY/FN_Zd7kOdJILQTAgOm1KxFROrcbIcAQ-gCLcB/s1600/OpenStack%2BMagnum.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="280" src="https://1.bp.blogspot.com/-xbOT-PKvXiA/VxaoJ1ZMTXI/AAAAAAAAjoY/FN_Zd7kOdJILQTAgOm1KxFROrcbIcAQ-gCLcB/s320/OpenStack%2BMagnum.png" width="320" /></a></div>
Magnum relies heavily on Heat for the orchestration part of the container clusters - called <b>bays</b>. Bays are instantiated based on pre-defined <b>bay models</b>, which set how the master and the other nodes should look like (flavor, image, etc) and which container orchestration engine (COE) should be used - among other possible configuration options. Current choices include Docker Swarm, Kubernetes and Mesos. The <a href="https://wiki.openstack.org/wiki/Magnum">Magnum homepage</a> gives a lot more details.<br />
<br />
At the beginning of November 2015, we started investigating the Magnum project in depth. At that time the project was functional but some of its requirements posed problems in our deployment:<br />
<ul style="text-align: left;">
<li>Dependency on OpenStack Neutron, something we had not yet deployed (we have nova-network since we started the cloud in 2012). Luckily we were working on it in parallel, and we got a functional control plane just in time. And as we use Nova Cells, we could enable Neutron in a dedicated cell where we would also enable Magnum, reusing the rest of the production infrastructure</li>
<li>Requirement on Neutron LBaaS, which we don't have. This is something we plan to try, but it is not obvious how to implement this currently due to the way the CERN network is structured. We made some changes to the Heat templates to remove this requirement</li>
</ul>
<div>
The other pre-requisite projects, such as Keystone, Glance and Heat were already in production in the CERN cloud.</div>
<div>
<br /></div>
<div>
But no real show stoppers and very quickly we got a prototype deployment. For a more detailed evaluation we initially chose 3 internal projects that cover the most common use cases:<br />
<ul style="text-align: left;">
<li><a href="https://about.gitlab.com/gitlab-ci/">GitLab CI</a>, a continuous integration service we use internally - it has integration with Docker, making it a perfect example of how to use a Docker Swarm cluster as a drop in replacement for a local Docker daemon</li>
<li>Infrastructure services, namely one of the critical services for the data movement between the multiple sites of the LHC Computing Grid (WLCG) - for a nice example of scaling a service by scaling its individual components</li>
<li><a href="http://jupyter.org/">Jupyter Notebooks</a> - a growing trend for end user analysis in different scientific communities, providing a browser based interactive session running in a remote container</li>
</ul>
<div>
In addition, we are also working with the European Union Horizon 2020 project <a href="https://www.indigo-datacloud.eu/">Indigo Datacloud</a> which is developing an open source data and computing platform targeted at scientific communities, deployable on multiple hardware and provisioned over hybrid, private or public, e-infrastructures. Using Magnum, we can provide the test resources for this project to the partners.</div>
<div>
<br />
For our users and resource managers, there are significant advantages of the Magnum approach:<br />
<ul>
<li>Native tools - anything that works with Docker will work talking to a Docker Swarm COE or kubectl with a Kubernetes cluster. This allows smaller physics sites to provide native Docker or Kubernetes while the larger sites provide Containers-as-a-Service on-demand. User applications written to work with Docker or Kubernetes can be used without modification against the provisioned resources.</li>
<li>Container engine agnostic - with our user community, there is a strong need for flexibility to allow different avenues to be explored. Magnum allows the IT department to offer Kubernetes, Docker Swarm and Mesos at a low cost within the same service at an affordable load for the support team. The users can then prototype different application approaches and select the best combination for them. Enforcing a central IT service decision on the end user community is never easy, especially where there are diverse user requirements being covered within a central cloud.</li>
<li>Accounting, quota and permissions remains within the existing framework. Thus, whether resources are used for containers or VMs is a choice for the project user. Capacity planning can be done by cores/RAM rather than segmentation of resources for container or VM resources. Access controls follow the existing admin/member structures for projects. </li>
<li>Elasticity - within the quota limits, containers can scale, with new bays as needed within the quota. This allows the resources to be allocated where there is a user need (and as importantly, shrunk when things are quiet)</li>
<li>Repairs - failures in the infrastructure (software or hardware) are looked after by the cloud support team. For the user, the workloads can be scheduled elsewhere. For the hardware repair teams, the operations can be performed in a consistent fashion in bulk rather than on a one-by-one basis. Infrastructure monitoring procedures are the same for VMs and containers.</li>
<li>The operating system support teams can provide reference images and follow up issues with the upstream providers. They can be confident that the image is based on supported configurations rather than ad-hoc builds. Rebuilding base images with the appropriate security patches can sometimes be delayed, raising the risk of incidents.</li>
</ul>
</div>
<div>
By the end of March, we had the use cases covered, and <a href="https://blueprints.launchpad.net/magnum/+spec/support-for-different-docker-storage-driver">the</a> <a href="https://blueprints.launchpad.net/magnum/+spec/allow-user-softwareconfig">few</a> <a href="https://blueprints.launchpad.net/magnum/+spec/create-trustee-user-for-each-bay">hick-ups</a> covered in blueprints or patches upstream, and had contributed for the missing bits in <a href="https://review.openstack.org/#/q/project:openstack/puppet-magnum">puppet</a> and <a href="https://blueprints.launchpad.net/magnum/+spec/magnum-installation-guide">documentation</a>. And with a service running on our production resources and thanks to keystone endpoint filtering, we could increase service usage by enabling it for individual projects. Today we have around 15 different projects using Magnum as a pilot service and the number keeps growing.<br />
<br />
In just a few months, we got Magnum up and running and it has proved to be a significant addition to the OpenStack cloud. Which makes us excited about what is coming next, including:<br />
<ul style="text-align: left;">
<li><a href="https://blueprints.launchpad.net/magnum/+spec/magnum-integrate-with-cinder">Integration with Cinder</a> - ready upstream, and we'll be trying it very soon</li>
<li><a href="https://blueprints.launchpad.net/magnum/+spec/add-magnum-to-rally">Magnum benchmarks in Rally</a> - we rely on Rally to make sure our cloud is performing as expected</li>
<li>Further integration with our local storage systems such as <a href="https://cernvm.cern.ch/portal/filesystem">CVMFS</a> and <a href="https://eos.web.cern.ch/">EOS</a> - relying on the ability to add <a href="https://blueprints.launchpad.net/magnum/+spec/allow-user-softwareconfig">site specific configurations</a> to the bay templates</li>
<li>Integration with Barbican - the recommended way to handle the required TLS certificates to talk to the native APIs of the orchestration engines, and the only option today to get Magnum in HA (though <a href="https://blueprints.launchpad.net/magnum/+spec/barbican-alternative-store">that's about to change</a>)</li>
<li><a href="https://github.com/openstack/magnum-ui">Integration with Horizon</a> - this will help as we expand the service into production to communities who are used to using the web interfaces</li>
</ul>
</div>
<div>
If you're interested in more details on the available container orchestration technologies or our usage of OpenStack Magnum, or simply want to see some fancy demos, check <a href="https://cds.cern.ch/record/2144886">our recent presentation</a> at a CERN Technical Forum.<br />
<h3 style="text-align: left;">
Acknowledgments</h3>
<ul style="text-align: left;">
<li>Mathieu Velten for his work on testing and adapting Magnum at CERN and contributions to Indigo DataCloud</li>
<li>Bertrand Noel for all his time spent researching existing container technologies</li>
<li>Spyros Trigazis, a fellow in the <a href="http://cern.ch/openlab">CERN OpenLab</a> collaboration with Rackspace, for all his work upstream both for features and documentation improvements</li>
<li>Jarek Polok for the <a href="http://docker.cern.ch/">CERN docker repository</a></li>
<li>The OpenStack Magnum team for their support and collaboration</li>
<li>All CERN users that helped us debug and set the service requirements</li>
</ul>
<h3>
References</h3>
</div>
</div>
<div>
<ul>
<li>Presentation on containers at the CERN Technical Forum - <a href="https://cds.cern.ch/record/2144886">https://cds.cern.ch/record/2144886</a></li>
<li>End user documentation at <a href="http://clouddocs.web.cern.ch/clouddocs/containers/index.html">http://clouddocs.web.cern.ch/clouddocs/containers/index.html</a></li>
<li>OpenStack superuser article at <a href="http://superuser.openstack.org/articles/openstack-magnum-on-the-cern-production-cloud">http://superuser.openstack.org/articles/openstack-magnum-on-the-cern-production-cloud</a></li>
</ul>
<div>
<br /></div>
</div>
</div>
Ricardo Rochahttp://www.blogger.com/profile/12954165425097243113noreply@blogger.com33tag:blogger.com,1999:blog-6032896665180559.post-58320755026036867162016-04-19T03:35:00.001-07:002016-06-25T08:59:39.744-07:00Deploying the new OpenStack EC2 API projectOpenStack has supported a subset of the EC2 API since the start of the project. This was originally built in to Nova directly. At CERN, we use this for a number of use cases where the experiments are running across both the on-premise and AWS clouds and would like a consistent API. A typical example of this is the <a href="https://www.google.ch/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwifsJWxt_7LAhWDORQKHbV7CVUQFggcMAA&url=https%3A%2F%2Fresearch.cs.wisc.edu%2Fhtcondor%2F&usg=AFQjCNFT940_-N3dwa_We7sdN9y21dLizQ&sig2=7v4yVZpgjGBPhwU9pBQD-Q&bvm=bv.118817766,d.bGg">HTCondor batch system</a> which can instantiate <a href="https://research.cs.wisc.edu/htcondor/HTCondorWeek2013/presentations/MillerT_EC2Tutorial.pptx">new workers according to demand in the queue</a> on the target cloud.<br />
<br />
With the <a href="https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#Upgrade_Notes_2">Kilo release</a>, this function was deprecated and has been removed in <a href="http://docs.openstack.org/releasenotes/nova/mitaka.html">Mitaka</a>. The functionality is now provided by the new <a href="http://git.openstack.org/cgit/openstack/ec2-api/">ec2-api</a> project which uses the public Nova APIs to provide an EC2 compatible interface.<br />
<br />
Given that CERN has the goal to upgrade to the latest OpenStack release in the production cloud before the next release is available, a migration to the ec2-api project was required before the deployment of Mitaka, due to be deployed at CERN in 2H 2016.<br />
<br />
The EC2 API project was easy to set up using the underlying information from Nova and a small database which is used to store some EC2 specific information such as tags.<br />
<br />
As described in <a href="https://www.subbu.org/blog/2013/07/openstack-is-not-cloud">Subbu's blog</a>, there are many parts needed before for an OpenStack API to become a service. By deploying using the CERN cloud, many aspects on identity, capacity planning, log handling, onboarding are covered by the existing infrastructure.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-OPF3gyUNrwA/VwdUQn4_o7I/AAAAAAAASr4/wNU6JxZoQ3k81gqyYhRo7A03AYMJzGPyg/s1600/subbu.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="283" src="https://2.bp.blogspot.com/-OPF3gyUNrwA/VwdUQn4_o7I/AAAAAAAASr4/wNU6JxZoQ3k81gqyYhRo7A03AYMJzGPyg/s320/subbu.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
From the CERN perspective, the key functions we need in addition to the code are</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul>
<li>Packaging - we work with the <a href="https://www.rdoproject.org/">RDO</a> distribution and the OpenStack <a href="https://wiki.openstack.org/wiki/Rpm-packaging">RPM-Packaging</a> project to produce a package for installation on our CentOS 7 controllers.</li>
<li>Configuration - Puppet provides us the configuration management for the CERN cloud. We are currently merging the <a href="https://github.com/cernops/puppet-ec2api">CERN Puppet EC2 API</a> modules to the <a href="https://github.com/openstack/puppet-ec2api">puppet-ec2api </a>project. The <a href="https://review.openstack.org/#/c/276103/">initial patch</a> is now in review.</li>
<li>Monitoring - each new project has a set of daemons to make sure are running smoothly. These have to be integrated into the site monitoring system.</li>
<li>Performance - we use the <a href="https://wiki.openstack.org/wiki/Rally">OpenStack Rally</a> project to continuously run functionality and performance tests, simulating a user. The EC2 support has been added in this <a href="https://review.openstack.org/#/c/147550/">review</a>.</li>
</ul>
<div>
The current steps are the end user testing and migration from the current service. Given that the ec2-api project can be run on a different port, the two services can be run in parallel for testing. Horizon would need to be modified to change the EC2 endpoint in the ec2rc.sh (which is downloaded from Compute->Account & Security->API Access).<br />
<br />
So far, the tests have been positive and further validation will be performed over the next few months to make sure that the migration has completed so there is no impact on the Mitaka upgrade.<br />
<br /></div>
<div>
<ul><ul>
</ul>
</ul>
</div>
<h3>
Acknowledgements</h3>
<div>
<ul>
<li>Wataru Takase (KEK) for his work on Rally</li>
<li>Marcos Fermin Lobo (CERN/Oviedo) for the packaging and configuration</li>
<li>Belmiro Moreira (CERN) for the necessary local CERN customisations in Nova</li>
<li>The folks from Cloudscaling/EMC for their implementation and support of the OpenStack EC2 API project</li>
</ul>
<h3>
References</h3>
<div>
<ul>
<li>CERN enduser documentation for EC2 at <a href="http://clouddocs.web.cern.ch/clouddocs/ec2tutorial/index.html">http://clouddocs.web.cern.ch/clouddocs/ec2tutorial/index.html</a></li>
</ul>
</div>
<div>
<br /></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com62tag:blogger.com,1999:blog-6032896665180559.post-1767596928739380492015-11-29T07:46:00.004-08:002015-12-10T10:49:19.494-08:00Our cloud in Kilo<br />
Following on from previous upgrades, CERN migrated the OpenStack cloud to Kilo during September to November. Along with the bug fixes, we are planning on exploiting the significant number of <a href="https://wiki.openstack.org/wiki/ReleaseNotes/Kilo">new features</a>, especially as related to performance tuning. The overall cloud architecture was covered at the Tokyo OpenStack summit video <a href="https://www.openstack.org/summit/tokyo-2015/videos/presentation/unveiling-cern-cloud-architecture">https://www.openstack.org/summit/tokyo-2015/videos/presentation/unveiling-cern-cloud-architecture</a>.<br />
<br />
As the LHC continues to run 24x7, these upgrades were done while the cloud was running and virtual machines were untouched.<br />
<br />
Previous upgrades have been described as below<br />
<ul>
<li>Juno - <a href="http://openstack-in-production.blogspot.fr/2015/05/our-cloud-in-juno.html">http://openstack-in-production.blogspot.fr/2015/05/our-cloud-in-juno.html</a></li>
<li>Icehouse - <a href="http://openstack-in-production.blogspot.fr/2014/11/our-cloud-in-icehouse.html">http://openstack-in-production.blogspot.fr/2014/11/our-cloud-in-icehouse.html</a></li>
<li>Havana - <a href="http://openstack-in-production.blogspot.fr/2014/02/our-cloud-in-havana.html">http://openstack-in-production.blogspot.fr/2014/02/our-cloud-in-havana.html</a></li>
</ul>
The staged approach was used again. While most of the steps went smoothly, a few problems were encountered.<br />
<ul>
<li>Cinder - we encountered the bug <a href="https://bugs.launchpad.net/cinder/+bug/1455726">https://bugs.launchpad.net/cinder/+bug/1455726</a> which led to a foreign key error. The cause appears to be related to UTF8. The patch (<a href="https://review.openstack.org/#/c/183814/">https://review.openstack.org/#/c/183814/</a>) was not completed so did not get included into the release. More details at the thread at <a href="http://lists.openstack.org/pipermail/openstack/2015-August/013601.html">http://lists.openstack.org/pipermail/openstack/2015-August/013601.html</a>.</li>
<li>Keystone - one of the configuration parameters for caches had changed syntax and this was not reflected in the configuration generated by Puppet. The symptoms were high load on the Keystone servers since caching was not enabled.</li>
<li>Glance - given the rolling upgrade on Glance, we took advantage of having virtualised the majority of the Glance server pool. This allows new resources to be brought online with a Juno configuration and the old ones deleted.</li>
<li>Nova - we upgraded the control plane services along with the QA compute nodes. With the versioned objects, we could stage the migration of the thousands of compute nodes so that we did not need to do all the updates at once. Puppet looked after the appropriate deployments of the RPMs.</li>
<ul>
<li>Following the upgrade, we had an outage of the metadata service for the OpenStack specific metadata. The EC2 metadata works fine. This is a cells related issue and we'll create a bug/blueprint for the fix.</li>
<li>The VM resize functions are giving errors during the execution. We're tracking this with the upstream developers.</li>
<ul>
<li><a href="https://bugs.launchpad.net/nova/+bug/1459758">https://bugs.launchpad.net/nova/+bug/1459758</a></li>
<li><a href="https://bugs.launchpad.net/nova/+bug/1446082">https://bugs.launchpad.net/nova/+bug/1446082</a></li>
</ul>
<li>We wanted to use the latest Nova NUMA features. We encountered a problem with cells and this feature, although it worked well in a non-cells cloud. This is being tracked in <a href="https://bugs.launchpad.net/nova/+bug/1517006">https://bugs.launchpad.net/nova/+bug/1517006</a>. We will use the new features for performance optimisation once these problems are resolved.</li>
<li>The <a href="http://www.danplanet.com/blog/2015/10/06/upgrades-in-nova-objects/">dynamic</a> migration of flavors was only partially successful. With the cells database having the flavors data in two places, the migration needed to be done simultaneously. We resolved this by forcing the migration of the flavors to the new endpoint,</li>
<li>The handling of ephemeral drives in Kilo seems to be different from Juno. The option <span style="font-family: monospace; font-size: 13.3333px;">default_ephemeral_format</span> now defaults to vfat, rather than ext3. The aim seems to have been to give vfat to Windows and ext4 to Linux but our environment does not follow this. This was reported by <a href="https://support.rc.nectar.org.au/news/18-09-2015/vfat-filesystem-secondary-ephemeral-disk-mnt-devvdb">Nectar</a> but we could not find any migration advice in the Kilo release notes. We have set the default to ext3 while we are working out the migration implications.</li>
<li>We're also working through a scaling problem for our most dynamic cells at <a href="https://bugs.launchpad.net/nova/+bug/1524114">https://bugs.launchpad.net/nova/+bug/1524114</a>. Here all VMs are being queried by the scheduler, not just the active ones. Since we create/delete hundreds of VMs an hour, there are large volumes of deleted VMs which made one query take longer than expected.</li>
</ul>
</ul>
<div>
Catching these cases with cells early is part of the work for the scope of the the Cell V2 project at <a href="https://wiki.openstack.org/wiki/Nova-Cells-v2">https://wiki.openstack.org/wiki/Nova-Cells-v2</a> to which we are contributing along with the BARC centre in Mumbai so that the cells configuration becomes the default (with only a single cell) and the upstream test cases are enhanced to validate the multi cell configuration.</div>
<div>
<br /></div>
<div>
As some of the hypervisors are still running Scientific Linux 6, we used the approach from GoDaddy to package the components using software collections. Details are available at <a href="https://github.com/krislindgren/openstack-venv-cent6">https://github.com/krislindgren/openstack-venv-cent6</a>. We used this for nova and ceilometer which are the agents installed on the hypervisors. The controllers were upgraded to CentOS 7 as part of the upgrade to Kilo.</div>
<div>
<br /></div>
<div>
Overall, getting to Kilo enables new features and includes bug fixes to reduce administration effort. Keeping up with new releases requires careful planning and sharing upstream activities such as the Puppet modules but has proven to be the best approach. With many of the CERN OpenStack team in the summit in Tokyo, we did not complete the upgrade before Liberty was released but this has been completed soon afterwards.</div>
<div>
<br /></div>
<div>
With the Kilo base in production, we are now ready to start work on the Nova network to Neutron migration, deployment of the new<a href="https://github.com/openstack/ec2-api"> EC2 API </a>project and enabling <a href="https://github.com/openstack/magnum">Magnum</a> for container native applications.</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com2tag:blogger.com,1999:blog-6032896665180559.post-82804163101164902762015-10-08T13:38:00.000-07:002015-10-08T13:53:27.004-07:00Scheduling and disabling CellsIn order to scale OpenStack Cloud Infrastructure at CERN, we were early to embrace an architecture that uses Cells. Cells is a Nova functionality that allows the partition a Cloud Infrastructure into smaller groups with independent control planes.<br />
<br />
For large deployments Cells have several advantages like: <br />
<div class="OutlineElement Ltr SCX74612536" style="font-family: Calibri, sans-serif; font-size: 8px; margin: 0px; padding: 0px;">
<div class="Paragraph SCX74612536" paraid="588217920" style="-webkit-nbsp-mode: normal !important; color: windowtext; font-size: 6pt; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
<br /></div>
</div>
<ul>
<li>single endpoint to users; </li>
<li>increase the availability and resilience of the Infrastructure; </li>
<li>avoid that Nova and external components (DBs, message brokers) reach their limits; </li>
<li>isolate different user cases; </li>
</ul>
However, cells also have some limitations. There are some nova features that don't work when running cells:<br />
<ul>
<li>Security Groups; </li>
<li>Manage aggregates on Top Cell; </li>
<li>Availability Zone support; </li>
<li>Server groups; </li>
<li>Cell scheduler limited functionality;</li>
</ul>
There has been many changes since we deployed our initial cells configuration two years ago. During the past months ,there have been a lot of work involving Cells, especially make sure that they are properly tested and developing CellsV2 that should be the default way to deploy Nova in the future. <br />
<br />
However, today when using Cells we continue to receive following welcome message :) <br />
<br />
"The cells feature of Nova is considered experimental by the OpenStack project because it receives much less testing than the rest of Nova. This may change in the future, but current deployers should be aware that the use of it in production right now may be risky." <br />
<br />
At CERN, we now have 26 children cells supporting the 130,000 cores across two data centres in a single cloud. Some cells are dedicated for the general use cases and others that are dedicated only to specific projects. <br />
<br />
In order to map projects to cells we developed a scheduler filter for the cell scheduler. <br />
<br />
<a href="https://github.com/cernops/nova/blob/cern-2014.2.2-2/nova/cells/filters/target_cell_project.py">https://github.com/cernops/nova/blob/cern-2014.2.2-2/nova/cells/filters/target_cell_project.py</a> <br />
<br />
The filter relies in two new values defined in nova.conf: "cells_default" and "cells_projects".<br />
<div>
<br />
<ul>
<li>"cells_default" contains the set of available cells to schedule instances if the project is not mapped to any specific cell; </li>
<li>"cells_projects" contains the mapping cell -> project for the specific use cases; </li>
</ul>
<br />
<div class="OutlineElement Ltr SCX74612536" style="margin: 0px; padding: 0px;">
<div class="Paragraph SCX74612536" paraid="1160012236" style="-webkit-nbsp-mode: normal !important; color: windowtext; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
<span style="font-family: Courier New, Courier, monospace;"><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">“</span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><span class="SpellingError SCX74612536" style="-webkit-nbsp-mode: normal !important; background-image: url(data:image/gif; background-position: 0% 100%; background-repeat: repeat no-repeat; border-bottom-color: transparent; border-bottom-style: solid; border-bottom-width: 1px; margin: 0px; padding: 0px;">nova.conf</span></span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">”</span><span class="EOP SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;"> </span></span></div>
</div>
<div class="OutlineElement Ltr SCX74612536" style="margin: 0px; padding: 0px;">
<div class="Paragraph SCX74612536" paraid="27186164" style="-webkit-nbsp-mode: normal !important; color: windowtext; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
<span style="font-family: Courier New, Courier, monospace;"><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><span class="SpellingError SCX74612536" style="-webkit-nbsp-mode: normal !important; background-image: url(data:image/gif; background-position: 0% 100%; background-repeat: repeat no-repeat; border-bottom-color: transparent; border-bottom-style: solid; border-bottom-width: 1px; margin: 0px; padding: 0px;">cells_default</span></span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">=</span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><span class="SpellingError SCX74612536" style="-webkit-nbsp-mode: normal !important; background-image: url(data:image/gif; background-position: 0% 100%; background-repeat: repeat no-repeat; border-bottom-color: transparent; border-bottom-style: solid; border-bottom-width: 1px; margin: 0px; padding: 0px;">cellA,cellB,cellC,cellD</span></span><span class="EOP SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;"> </span></span></div>
</div>
<div class="OutlineElement Ltr SCX74612536" style="margin: 0px; padding: 0px;">
<div class="Paragraph SCX74612536" paraid="1379603675" style="-webkit-nbsp-mode: normal !important; color: windowtext; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
<span style="font-family: Courier New, Courier, monospace;"><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><span class="SpellingError SCX74612536" style="-webkit-nbsp-mode: normal !important; background-image: url(data:image/gif; background-position: 0% 100%; background-repeat: repeat no-repeat; border-bottom-color: transparent; border-bottom-style: solid; border-bottom-width: 1px; margin: 0px; padding: 0px;">cells_projects</span></span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">=</span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><span class="SpellingError SCX74612536" style="-webkit-nbsp-mode: normal !important; background-image: url(data:image/gif; background-position: 0% 100%; background-repeat: repeat no-repeat; border-bottom-color: transparent; border-bottom-style: solid; border-bottom-width: 1px; margin: 0px; padding: 0px;">cellE</span></span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">:<project_uuid1>;<project_uuid2>,</span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><span class="SpellingError SCX74612536" style="-webkit-nbsp-mode: normal !important; background-image: url(data:image/gif; background-position: 0% 100%; background-repeat: repeat no-repeat; border-bottom-color: transparent; border-bottom-style: solid; border-bottom-width: 1px; margin: 0px; padding: 0px;">cellF</span></span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">:</span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US"><</span><span class="TextRun SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;" xml:lang="EN-US">project_uuid3></span><span class="EOP SCX74612536" style="-webkit-nbsp-mode: normal !important; line-height: 18px; margin: 0px; padding: 0px;"> </span></span></div>
</div>
<div class="OutlineElement Ltr SCX74612536" style="font-family: Calibri, sans-serif; font-size: 8px; margin: 0px; padding: 0px;">
<div class="Paragraph SCX74612536" paraid="1541311007" style="-webkit-nbsp-mode: normal !important; color: windowtext; font-size: 6pt; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
<br /></div>
</div>
<br />
For example, when an instance belonging to "project_uuid2" is created, it's schedule to "cellE". But, if the instance belongs to "project_uuid4" it's schedule to one of the default cells ("cellA", "cellB", "cellC", "cellD"). <br />
<br />
One of the problems when using cells is that is not possible to disable them from the scheduler. <br />
<br />
With this scheduler filter we can achieve this. To disable a cell we just need to remove it from the "cells_default" or" cells_projects" list. Disabling a cell means that it will not be possible to create new instances on it, however it is still available to perform operations like restart, resize, delete, ...<br />
<br />
<br />
These experiences will be discussed in the upcoming summit in Tokyo with the deep dive into the CERN OpenStack deployment (<a href="https://mitakadesignsummit.sched.org/event/f929ea7ee625dadcc16888cb33984dad#.VhbHu3pCrWI">https://mitakadesignsummit.sched.org/event/f929ea7ee625dadcc16888cb33984dad#.VhbHu3pCrWI</a>), at the Ops meetup (<a href="https://etherpad.openstack.org/p/TYO-ops-meetup">https://etherpad.openstack.org/p/TYO-ops-meetup</a>) and Nova design sessions (<a href="https://mitakadesignsummit.sched.org/overview/type/nova#.VhbHaXpCrWI">https://mitakadesignsummit.sched.org/overview/type/nova#.VhbHaXpCrWI</a>)<br />
<br />
<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/04545390781777747883noreply@blogger.com2tag:blogger.com,1999:blog-6032896665180559.post-10418260329553463922015-09-26T07:23:00.000-07:002017-02-07T11:44:03.416-08:00EPT, Huge Pages and Benchmarking<br />
Having reported that <a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">EPT has a negative influence</a> on the High Energy Physics standard benchmark <a href="http://iopscience.iop.org/1742-6596/219/5/052009/pdf/1742-6596_219_5_052009.pdf">HepSpec06</a>, we have started the deployment of those settings across the CERN OpenStack cloud,<br />
<ul>
<li>Setting the flag in /etc/modprobe.d/kvm_intel.conf to off</li>
<li>Waiting for the work on each guest to finish after stopping new VMs on the hypervisor</li>
<li>Changing the flag and reloading the module</li>
<li>Enabling new work for the hypervisor</li>
</ul>
<div>
According to the HS06 tests, this should lead to a reasonable performance improvement based on the results of the benchmark and tuning. However, certain users reported significantly worse performance than previously. In particular, some workloads showed significant differences in the following before and after characteristics.</div>
<div>
<br /></div>
<div>
Before the workload was primarily CPU bound, spending most of its time in user space. CERN applications have to process significant amounts of data so it is not always possible to ensure 100% utilisation but the aim is to provide the workload with user space CPU.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-sj8QOJji9JM/VeiN65TR6nI/AAAAAAAALok/FBbteU68mC4/s1600/with%2Bept%2Bon.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="170" src="https://3.bp.blogspot.com/-sj8QOJji9JM/VeiN65TR6nI/AAAAAAAALok/FBbteU68mC4/s640/with%2Bept%2Bon.png" width="640" /></a></div>
<div>
<br /></div>
<div>
When EPT was turned off. some selected hypervisors showed a very difference performance profile. A major increase in non-user load and a reduction in the throughput for the experiment workloads. However, this effect was not observed on the servers with AMD processors.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-wHjp7E08Gzg/VeiOIxPrfQI/AAAAAAAALos/Sissxfx2-qY/s1600/with%2Bept%2Boff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="170" src="https://1.bp.blogspot.com/-wHjp7E08Gzg/VeiOIxPrfQI/AAAAAAAALos/Sissxfx2-qY/s640/with%2Bept%2Boff.png" width="640" /></a></div>
<div>
<br /></div>
<div>
With tools such as perf, we were able to trace the time down to handling the TLB misses. Perf gives</div>
<div>
<br />
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace;">78.75% [kernel] [k] _raw_spin_lock<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace;">6.76% [kernel] [k] set_spte<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace;">1.97% [kernel] [k] memcmp<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace;">0.58% [kernel] [k] vmx_vcpu_run<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace;">0.46% [kernel] [k] ksm_docan<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace;">0.44% [kernel] [k] vcpu_enter_guest</span><o:p></o:p></div>
<br />
<div class="MsoNormal">
The process behind the _raw_spin_lock is qemu-kvm.</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<br />
<div class="MsoNormal">
Using systemtap kernel backtraces, we see mostly page faults and spte_* commands (shadow page table updates)</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
Both of these should not be necessary if you have hardware support for address translation: aka EPT.<br />
<br /></div>
There may be specific application workloads where the EPT setting was non optimal. In the worst case, the performance was several times slower. EPT/NPT increases the cost of doing page table walks when the page is not cached in the TLB. This document shows how processors can speed up page walks - <a href="http://www.cs.rochester.edu/~sandhya/csc256/seminars/vm_yuxin_yanwei.pdf">http://www.cs.rochester.edu/~sandhya/csc256/seminars/vm_yuxin_yanwei.pdf</a> and AMD includes a page walk cache in their processor which speeds up the walking of pages as described in this paper <a href="http://vglab.cse.iitd.ac.in/~sbansal/csl862-virt/readings/p26-bhargava.pdf">http://vglab.cse.iitd.ac.in/~sbansal/csl862-virt/readings/p26-bhargava.pdf</a><br />
<br />
In other words, EPT slows down HS06 results when there are small pages involved because the HS06 benchmarks miss the TLB a lot. NPT doesn't slow it down because AMD has a page walk cache to help speed up finding the pages when it's not in the TLB. EPT comes good again when we have large pages because it rarely results in a TLB miss. So, HS06 is probably representative of most of the job types, but the is a small share of jobs which are different and triggered the above-mentioned problem.<br />
<br /></div>
<div class="MsoNormal">
However, we have 6% overhead compared to previous runs due to EPT on for the benchmark as mentioned in the <a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">previous blog</a>. Mitigating the EPT overheads following the comments on the <a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">previous blog</a>, we looked into using dedicated Huge Pages. Our hypervisors run CentOS 7 and thus support both transparent huge pages and huge pages. Transparent huge pages performs a useful job under normal circumstances but are opportunistic in nature. They are also limited to 2MB and cannot use the 1GB maximum size.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
We tried setting the default huge page to 1G using the Grub cmdline configuration.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat /sys/kernel/mm/transparent_hugepage/enabled</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">[always] madvise never</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat /boot/grub2/grub.cfg | grep hugepage</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">linux16 /vmlinuz-3.10.0-229.11.1.el7.x86_64 root=UUID=7d5e2f2e-463a-4842-8e11-d6fac3568cf4 ro rd.md.uuid=3ff29900:0eab9bfa:ea2a674d:f8b33550 rd.md.uuid=5789f86e:02137e41:05147621:b634ff66 console=tty0 nodmraid crashkernel=auto crashkernel=auto rd.md.uuid=f6b88a6b:263fd352:c0c2d7e6:2fe442ac vconsole.font=latarcyrheb-sun16 vconsole.keymap=us LANG=en_US.UTF-8 default_hugepagesz=1G hugepagesz=1G hugepages=55 transparent_hugepage=never</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat /sys/module/kvm_intel/parameters/ept</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Y</span><br />
<div>
<br /></div>
<div>
It may also be advisable to disable tuned for the moment until the <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1189868">bug #1189868 </a>is resolved.</div>
</div>
<div class="MsoNormal">
<br />
We also configured the XML manually to include the necessary huge pages. This will be available as a flavor or image option when we upgrade to Kilo in a few weeks.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <memoryBacking></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <hugepages></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <page size="1" unit="G" nodeset="0-1"/></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </hugepages></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </memoryBacking></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The hypervisor was configured with huge pages enabled. However, we saw a problem with the distribution of huge pages across the NUMA nodes.</div>
<div class="MsoNormal">
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext"><br /></span></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ cat /sys/devices/system/node/node*/meminfo | fgrep Huge</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 0 AnonHugePages: 311296 kB</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 0 HugePages_Total: 29</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 0 HugePages_Free: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 0 HugePages_Surp: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 1 AnonHugePages: 4096 kB</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 1 HugePages_Total: 31</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 1 HugePages_Free: 2</span><br />
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext" style="font-family: "courier new" , "courier" , monospace;"></span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 1 HugePages_Surp: 0</span><br />
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext"><br /></span></span>
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext">This shows that the pages were not evenly distributed across the NUMA nodes., which would lead to subsequent performance issues. The suspicion is that the Linux boot up sequence led to some pages being used and this made it difficult to find contiguous blocks of 1GB for the huge pages. This led us to deploy 2MB pages rather than 1GB for the moment, while may not be the optimum setting allows better optimisations than the 4K settings and still gives some potential for KSM to benefit. These changes had a positive effect as the monitoring below shows when the reduction in system time.</span></span><br />
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext"><br /></span></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-wIlCMUGqfVU/VgVfO-1wQMI/AAAAAAAALqQ/iHlzxdBi70g/s1600/turnonept.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="203" src="https://2.bp.blogspot.com/-wIlCMUGqfVU/VgVfO-1wQMI/AAAAAAAALqQ/iHlzxdBi70g/s640/turnonept.png" width="640" /></a></div>
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext"><br /></span></span>
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext"><br /></span></span>
At the OpenStack summit in Tokyo, we'll be having a session on Hypervisor Tuning so people are welcome to bring their experiences along and share the various options. Details of the session will appear at <a href="https://etherpad.openstack.org/p/TYO-ops-meetup">https://etherpad.openstack.org/p/TYO-ops-meetup</a>.<br />
<br />
Contributions from Ulrich Schwickerath and Arne Wiebalck (CERN) and Sean Crosby (University of Melbourne) have been included in this article along with the help of the LHC experiments to validate the configuration.<br />
<span style="font-family: "arial" , sans-serif; font-size: 10.5pt; position: relative; top: -1.5pt;"><span class="diffcontext"><br /></span></span>
<br />
<h2>
References</h2>
<div>
<ul>
<li>OpenStack documentation now at <a href="http://docs.openstack.org/admin-guide/compute-adv-config.html">http://docs.openstack.org/admin-guide/compute-adv-config.html</a></li>
<li>Previous analysis for EPT at <a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html</a></li>
<li>Red Hat blog on Huge Pages at <a href="http://redhatstackblog.redhat.com/2015/09/15/driving-in-the-fast-lane-huge-page-support-in-openstack-compute/">http://redhatstackblog.redhat.com/2015/09/15/driving-in-the-fast-lane-huge-page-support-in-openstack-compute/</a></li>
<li>Mirantis blog on Huge Pages at <a href="https://www.mirantis.com/blog/mirantis-openstack-7-0-nfvi-deployment-guide-huge-pages/">https://www.mirantis.com/blog/mirantis-openstack-7-0-nfvi-deployment-guide-huge-pages/</a></li>
<li>VMWare paper on EPT at <a href="https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf">https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf</a></li>
<li>Academic studies of the overheads and algorithms of EPT and NPT (AMD's technology) at <a href="http://www.cs.rochester.edu/~sandhya/csc256/seminars/vm_yuxin_yanwei.pd">http://www.cs.rochester.edu/~sandhya/csc256/seminars/vm_yuxin_yanwei.pd</a>f and <a href="http://vglab.cse.iitd.ac.in/~sbansal/csl862-virt/readings/p26-bhargava.pdf">http://vglab.cse.iitd.ac.in/~sbansal/csl862-virt/readings/p26-bhargava.pdf</a></li>
</ul>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com4tag:blogger.com,1999:blog-6032896665180559.post-1654905820499289192015-08-05T08:58:00.002-07:002017-02-07T11:45:06.849-08:00Tuning hypervisors for High Throughput ComputingOver the past set of blogs, we've looked at a number of different options for tuning High Energy Physics workloads in a KVM environment such as the CERN OpenStack cloud.<br />
<br />
This is a summary of the findings using the <a href="http://iopscience.iop.org/1742-6596/219/5/052009/pdf/1742-6596_219_5_052009.pdf">HEPSpec 06</a> benchmark on KVM and a comparison with Hyper-V for the same workload.<br />
<br />
For KVM on this workload, we saw a degradation in performance on large VMs.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-oETOS9_n9pE/VcIvOlPIpFI/AAAAAAAALkI/tSwDbVB3ebo/s1600/vmoverhead.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="394" src="https://4.bp.blogspot.com/-oETOS9_n9pE/VcIvOlPIpFI/AAAAAAAALkI/tSwDbVB3ebo/s640/vmoverhead.png" width="640" /></a></div>
<br />
Results for other applications may vary so each option should be verified for the target environment. The percentages from our optimisations are not necessarily additive but give an indication of the performance improvements to be expected. After tuning, we saw around 5% overhead from the following improvements.<br />
<br />
<table border="1">
<tbody>
<tr><th>Option</th><th>Improvement</th><th>Comments</th>
</tr>
<tr><td><a href="http://openstack-in-production.blogspot.fr/2015/08/openstack-cpu-topology-for-high.html">CPU topology</a></td><td align="right">~0</td><td>The primary focus for this function was not for performance so result is as expected</td></tr>
<tr><td><a href="http://openstack-in-production.blogspot.fr/2015/08/cpu-model-selection-for-high-throughput.html">Host Model</a></td><td align="right">4.1-5.2%</td><td>Some impacts on operations such as live migration</td></tr>
<tr><td><a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">Turn EPT off</a></td><td align="right">6%</td><td>Open bug report for CentOS 7 guest on CentOS 7 hypervisor</td></tr>
<tr><td><a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">Turn KSM off</a></td><td align="right">0.9%</td><td>May lead to an increase in memory usage</td></tr>
<tr><td><a href="http://openstack-in-production.blogspot.fr/2015/08/numa-and-cpu-pinning-in-high-throughput.html">NUMA in guest</a></td><td align="right">~9%</td><td>Needs Kilo or later to generate this with OpenStack</td></tr>
<tr><td><a href="http://openstack-in-production.blogspot.fr/2015/08/numa-and-cpu-pinning-in-high-throughput.html">CPU Pinning</a><span id="goog_275387583"></span><span id="goog_275387584"></span><a href="https://www.blogger.com/"></a></td><td align="right">~3%</td><td>Needs Kilo or later (cumulative on top of NUMA)</td></tr>
</tbody></table>
<br />
Different applications will see a different range of improvements (or even that some of these options degrade performance). Experiences from other workload tuning would be welcome.<br />
<br />
One of the things that led us to focus on KVM tuning was the comparison with Hyper-V. At CERN, we made an early decision to run a multi-hypervisor cloud building on the work by <a href="http://cloudbase.it/">cloudbase.it</a> and Puppet on Windows to share the deployment scripts for both CentOS and Windows hypervisors. This allows us to direct appropriate workloads to the best hypervisor for the job.<br />
<br />
One of the tests when we saw a significant overhead on the default KVM configuration was to compare the performance overheads for a Linux configuration on Hyper-V. Interestingly, Hyper-V achieved better performance without tuning compared to the configurations with KVM. Equivalent tests on Hyper-V showed<br />
<div>
</div>
<ul style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
<li>4 VMs 8 cores: 0.8% overhead compared to bare metal </li>
<li>1 VM 32 cores: 3.3% overhead compared to bare metal</li>
</ul>
<div>
These performance results allowed us to focus on the potential areas for optimisation, that we needed to tune the hypervisor rather than a fundamental problem with virtualisation (with the results above for NUMA and CPU pinning)</div>
<div>
<br />
The Hyper-V configuration pins each core to the underlying NUMA socket which is similar to how the Kilo NUMA tuning sets KVM up.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-j2lVBm-CYsc/VcItM_Y8EeI/AAAAAAAALj0/OjO1J11UwT0/s1600/hv001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="604" src="https://3.bp.blogspot.com/-j2lVBm-CYsc/VcItM_Y8EeI/AAAAAAAALj0/OjO1J11UwT0/s640/hv001.png" width="640" /></a></div>
<span id="goog_426474528"></span><span id="goog_426474529"></span><br />
and<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-HjxewvXb4ik/VcItTnpFPtI/AAAAAAAALj8/6leq_AWXJX0/s1600/hv001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="604" src="https://2.bp.blogspot.com/-HjxewvXb4ik/VcItTnpFPtI/AAAAAAAALj8/6leq_AWXJX0/s640/hv001.png" width="640" /></a></div>
<br />
This gives the Linux guest configuration as seen from the guest running on a Hyper-V hypervisor<br />
<br />
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># numactl --hardware<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">available: 2 nodes (0-1)<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 0 size: 28999 MB<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 0 free: 27902 MB<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 1 size: 29000 MB<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 1 free: 28027 MB<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node distances:<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">node 0 1<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 0: 10 20</span></div>
<div class="MsoNormal">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 1: 20 10</span><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Thanks to the QEMU discuss mailing list and to the other team members who helped understand the issue (Sean Crosby (University of Melbourne) and Arne Wiebalck, Sebastian Bukowiec and Ulrich Schwickerath (CERN))<br />
<br />
<h2>
References</h2>
<div>
<ul>
<li>Recent 2017 documentation is now at h<a href="ttp://docs.openstack.org/admin-guide/compute-adv-config.html">ttp://docs.openstack.org/admin-guide/compute-adv-config.html</a></li>
</ul>
</div>
</div>
</div>
<br />
<br />Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com8tag:blogger.com,1999:blog-6032896665180559.post-16155190979814278122015-08-03T04:48:00.001-07:002017-02-07T01:43:02.437-08:00NUMA and CPU Pinning in High Throughput Computing<br />
CERN's OpenStack cloud runs the Juno release on mainly CentOS 7 hypervisors.<br />
Along with previous tuning options described in this blog which can be used on Juno, a number of further improvements have been delivered in Kilo.<br />
<br />
Since this release will be installed at CERN during the autumn, we had to configure standalone KVM configurations to test the latest features, in particular around NUMA and CPU pinning.<br />
<br />
<div>
<a href="https://en.wikipedia.org/wiki/Non-uniform_memory_access">NUMA</a> features have been appearing in more recent processors that means memory accesses are no longer uniform. Rather than a single large pool of memory accessed by the processors, the performance of the memory access varies according to whether the memory is local to the processor.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ryStuylAtk0/Vb3Wlg32pHI/AAAAAAAALi4/mbyDnaej6v8/s1600/NUMA.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="293" src="https://4.bp.blogspot.com/-ryStuylAtk0/Vb3Wlg32pHI/AAAAAAAALi4/mbyDnaej6v8/s400/NUMA.png" width="400" /></a>(<a href="http://frankdenneman.nl/2015/02/27/memory-deep-dive-numa-data-locality/">Frank Denneman</a><span id="goog_1027735052"></span><span id="goog_1027735053"></span><a href="https://www.blogger.com/"></a>)</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
A typical case above is where VM 1 is running on CPU 1 and needs a page of memory to be allocated. It is important that the memory allocated by the underlying hypervisor is the fastest access possible for the VM1 to access in future. Thus, the guest VM kernel needs to be aware of the underlying memory architecture of the hypervisor.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The NUMA configuration of a machine can be checked using lscpu. This shows two NUMA nodes on CERN's standard server configurations (two processors with 8 physical cores and SMT enabled)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lscpu</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Architecture: x86_64</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU op-mode(s): 32-bit, 64-bit</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Byte Order: Little Endian</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU(s): 32</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">On-line CPU(s) list: 0-31</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Thread(s) per core: 2</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Core(s) per socket: 8</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Socket(s): 2</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node(s): 2</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Vendor ID: GenuineIntel</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU family: 6</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Model: 62</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Model name: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Stepping: 4</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU MHz: 2257.632</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">BogoMIPS: 5206.18</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Virtualization: VT-x</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L1d cache: 32K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L1i cache: 32K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L2 cache: 256K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L3 cache: 20480K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node0 CPU(s): 0-7,16-23</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node1 CPU(s): 8-15,24-31</span></div>
<div>
<br /></div>
<div>
Thus, cores 0-7 and 16-23 are attached to the first NUMA node with the others on the second. The two ranges come from SMT. VMs however see a single NUMA node.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node0 CPU(s): 0-31</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h3 style="clear: both; text-align: left;">
First Approach - numad</h3>
<div>
The VMs on the CERN cloud are distributed across different sizes. Since there is a mixture of VM sizes, NUMA has a correspondingly varied influence.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-oLqiBX_ImXA/Vb3gTg2RTNI/AAAAAAAALjI/6Uxp6PNbXDA/s1600/vmdistribution.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="354" src="https://2.bp.blogspot.com/-oLqiBX_ImXA/Vb3gTg2RTNI/AAAAAAAALjI/6Uxp6PNbXDA/s640/vmdistribution.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
Linux provides the numad daemon which provides some automated balancing of NUMA workloads to move memory near to the processor where the thread is running.</div>
<div>
<br /></div>
<div>
In the case of 8 core VMs, numad on the hypervisor provided a performance gain of 1.6%. However, the effects for larger VMs was much less significant. Looking at the performance for running 4x8 core VMs versus 1x32 core VM, there was significantly more overhead for the large VM case.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-eXmMLu0x838/Vb3liHzchAI/AAAAAAAALjg/g4hMP_IL0Dc/s1600/vmoverhead.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="394" src="https://2.bp.blogspot.com/-eXmMLu0x838/Vb3liHzchAI/AAAAAAAALjg/g4hMP_IL0Dc/s640/vmoverhead.png" width="640" /></a></div>
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h3 style="clear: both; text-align: left;">
Second approach - expose NUMA to guest VM</h3>
<div>
This can be done using appropriate KVM directives. With OpenStack Kilo, these will be possible via the flavors extra specs and image properties. In the meanwhile, we configured the hypervisor with the following XML for libvirt.</div>
<div>
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><cpu mode='host-passthrough'></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><numa></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><cell id='0' cpus='0-7' memory='16777216'/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><cell id='1' cpus='16-23' memory='16777216'/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><cell id='2' cpus='8-15' memory='16777216'/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><cell id='3' cpus='24-31' memory='16777216'/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></numa></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></cpu></span></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
In an ideal world, there would be two cells defined (0-7,16-23 and 8-15,24-31) but KVM currently does not support non-contiguous ranges on CentOS 7 <span style="color: red;">[1]</span>. The guests see the configuration as follows</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lscpu</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Architecture: x86_64</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU op-mode(s): 32-bit, 64-bit</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Byte Order: Little Endian</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU(s): 32</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">On-line CPU(s) list: 0-31</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Thread(s) per core: 2</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Core(s) per socket: 8</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Socket(s): 2</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node(s): 4</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Vendor ID: GenuineIntel</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU family: 6</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Model: 62</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Model name: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Stepping: 4</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CPU MHz: 2593.750</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">BogoMIPS: 5187.50</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Hypervisor vendor: KVM</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Virtualization type: full</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L1d cache: 32K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L1i cache: 32K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">L2 cache: 4096K</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node0 CPU(s): 0-7</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node1 CPU(s): 8-15</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node2 CPU(s): 16-23</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">NUMA node3 CPU(s): 24-31</span></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
With this approach and turning off numad on the hypervisor, the performance of the large VM improved by 9%.</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
We also investigated the numatune options but these did not produce a significant improvement.</div>
<div class="separator" style="clear: both;">
<br /></div>
<h3 style="clear: both;">
Third Approach - Pinning CPUs</h3>
<div class="separator" style="clear: both; text-align: left;">
From the hypervisor's perspective, the virtual machine appears as a single process which needs to be scheduled on the available CPUs. While the NUMA configuration above means that memory access from the processor will tend to be local, the hypervisor may then choose to place the next scheduled clock tick on a different processor. While this is useful in the case of hypervisor over-commit, for a CPU bound application, this leads to less memory locality.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
With Kilo, it will be possible to pin a virtual core to a physical one. The same was done using the hypervisor XML as for NUMA.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><cputune></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><vcpupin vcpu="0" cpuset="0"/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><vcpupin vcpu="1" cpuset="1"/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><vcpupin vcpu="2" cpuset="2"/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><vcpupin vcpu="3" cpuset="3"/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><vcpupin vcpu="4" cpuset="4"/></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><vcpupin vcpu="5" cpuset="5"/></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">...</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
This will mean that the virtual core #1 is always run on the physical core #1.</div>
<div class="separator" style="clear: both; text-align: left;">
Repeating the large VM test provided a further 3% performance improvement.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The exact topology has been set in a simple fashion. Further investigation on getting exact mappings between thread siblings is needed to get the most of out of the tuning.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The impact on smaller VMs (8 and 16 core) is also needing to be studied. Optimising for one use case has a risk that other scenarios may be affected. Custom configurations for particular topologies of VMs increases the operations effort to run a cloud at scale. While the changes should be positive, or at minimum neutral, this needs to be verified.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h3 style="clear: both; text-align: left;">
Summary</h3>
<div>
Exposing the NUMA nodes and using CPU pinning has reduced the large VM overhead with KVM from 12.9% to 3.5%. When the features are available in OpenStack Kilo, these can be deployed by setting up the appropriate flavors with the additional pinning and NUMA descriptions for the different hardware types so that large VMs can be run at a much lower overhead.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
This work was in collaboration with Sean Crosby (University of Melbourne) and Arne Wiebalck and Ulrich Schwickerath (CERN).</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
Previous blogs in this series are<br />
<ul>
<li>CPU topology - <a href="http://openstack-in-production.blogspot.fr/2015/08/openstack-cpu-topology-for-high.html" style="text-decoration: none;">http://openstack-in-production.blogspot.fr/2015/08/openstack-cpu-topology-for-high.html</a></li>
<li>CPU model selection - <a href="http://openstack-in-production.blogspot.fr/2015/08/cpu-model-selection-for-high-throughput.html">http://openstack-in-production.blogspot.fr/2015/08/cpu-model-selection-for-high-throughput.html</a></li>
<li>KSM and EPT - <a href="http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html">http://openstack-in-production.blogspot.fr/2015/08/ept-and-ksm-for-high-throughput.html</a></li>
</ul>
<h3>
Updates</h3>
<div>
<span style="color: red;">[1]</span> RHEV does support this with the later QEMU rather than the default in CentOS 7 (http://cbs.centos.org/repos/virt7-kvm-common-testing/x86_64/os/Packages/, version 2.1.2)</div>
<h3>
References</h3>
<ul>
<li>Detailed presentation on the optimisations - <a href="https://indico.cern.ch/event/384358/contributions/909247/">https://indico.cern.ch/event/384358/contributions/909247/</a></li>
<li>Red Hat's tuning guide - <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-NUMA-NUMA_and_libvirt.html">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-NUMA-NUMA_and_libvirt.html</a></li>
<li>Stephen Gordon's description of the Kilo features -<a href="http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/">http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/</a></li>
<li>NUMA memory architecture - <a href="http://frankdenneman.nl/2015/02/27/memory-deep-dive-numa-data-locality/">http://frankdenneman.nl/2015/02/27/memory-deep-dive-numa-data-locality/</a></li>
<li>OpenStack memory placement - <a href="https://wiki.openstack.org/wiki/VirtDriverGuestCPUMemoryPlacement">https://wiki.openstack.org/wiki/VirtDriverGuestCPUMemoryPlacement</a></li>
<li>Fedora work - <a href="http://bderzhavets.blogspot.fr/2015/07/cpu-pinning-and-numa-topology-on-rdo_31.html">http://bderzhavets.blogspot.fr/2015/07/cpu-pinning-and-numa-topology-on-rdo_31.html</a></li>
</ul>
Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com4tag:blogger.com,1999:blog-6032896665180559.post-7816036554123025252015-08-02T10:35:00.002-07:002015-09-22T10:35:45.078-07:00EPT and KSM for High Throughput ComputingAs part of the analysis of CERN's compute intensive workload in a virtualised infrastructure, we have been examining various settings of KVM to tune the performance.<br />
<br />
<h3>
EPT</h3>
<a href="https://en.wikipedia.org/wiki/Second_Level_Address_Translation">EPT</a> is an Intel technology which provides hardware assist for virtualisation and is one of the options as part of the Intel KVM driver in Linux. This is turned on by default but can be controlled using the options on the KVM driver. The driver can then be reloaded as long as no qemu-kvm processes are running.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"># cat /etc/modprobe.d/kvm_intel.conf</span><br />
<span style="font-family: Courier New, Courier, monospace;">options kvm_intel ept=0</span><br />
<span style="font-family: Courier New, Courier, monospace;"># modprobe -r kvm_intel</span><br />
<span style="font-family: Courier New, Courier, monospace;"># modprobe kvm_intel</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
In past <a href="https://gridka-school.scc.kit.edu/2011/downloads/CloudComputing_070911_Ulrich_Schwickerath.pdf">studies</a>, EPT has had a negative performance impact on High Energy Physics applications. With recent changes in processor architecture, this was re-tested as follows.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-uNVANyDKbjc/Vb3MPNzZyLI/AAAAAAAALic/gkj7A_07F5A/s1600/ept%2Bsetting.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="380" src="http://2.bp.blogspot.com/-uNVANyDKbjc/Vb3MPNzZyLI/AAAAAAAALic/gkj7A_07F5A/s640/ept%2Bsetting.png" width="640" /></a></div>
<br />
This is a 6% performance improvement with EPT off. This seems surprising as the functions are intended to improve virtualisation performance rather than reduce it.<br />
<br />
The CERN configuration uses hypervisors running CentOS 7 and guests running Scientific Linux CERN 6. With this configuration, EPT can be turned off without problems but a recent test with CentOS 7 guests has shown that this functionality has an issue which has been reported upstream. Only one CPU is recognised and the rest are reported as being unresponsive.<br />
<h3>
KSM</h3>
<div>
<a href="https://en.wikipedia.org/wiki/Kernel_same-page_merging">Kernel same-page merging</a> is a technology which finds common memory pages inside a linux system and merges the pages so there is only a single copy, saving memory resources. In the event of one of the copies being updated, a new copy is created so the function is transparent to the processes on the system.</div>
<div>
<br /></div>
<div>
For hypervisors, this can be very beneficial where multiple guests are running with the same level of operating system. However, there is an overhead due to the scanning process which may cause the applications to run more slowly. </div>
<div>
<br /></div>
<div>
We benchmarked 4 VMs, each 8 cores, running the same operating system levels. The results were that KSM causes an overhead of around 1%.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-DyIHLVDp_II/Vb3QrbRc38I/AAAAAAAALio/5cRbenlbbXM/s1600/ksm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="382" src="http://4.bp.blogspot.com/-DyIHLVDp_II/Vb3QrbRc38I/AAAAAAAALio/5cRbenlbbXM/s640/ksm.png" width="640" /></a></div>
<div>
<br /></div>
<br />
To turn KSM off, the ksmtuned daemon should be stopped.<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">systemctl disable ksmtuned</span><br />
<br />
The ksmd kernel thread still seems to run but does not use any CPU resources. Following the change, it is important to verify that there is still sufficient memory on the hypervisor since not merging the pages could cause an increase in memory usage and lead to swapping (which is a very significant performance impact)<br />
<br />
<span style="font-family: inherit;"><span style="background-color: white; color: #666666; line-height: 18.4799995422363px;">This work was in collaboration with Sean Crosby (University of Melbourne) and Arne Wiebalck and </span><span style="background-color: white; color: #666666; line-height: 18.4799995422363px;">Ulrich Schwickerath</span><span style="background-color: white; color: #666666; line-height: 18.4799995422363px;"> </span><span style="background-color: white; color: #666666; line-height: 18.4799995422363px;"> (CERN).</span></span><br />
<span style="background-color: white; color: #666666; font-family: inherit; line-height: 18.4799995422363px;"><br /></span>
Previous blogs in this series are<br />
<ul>
<li>CPU topology - <a href="http://openstack-in-production.blogspot.fr/2015/08/openstack-cpu-topology-for-high.html">http://openstack-in-production.blogspot.fr/2015/08/openstack-cpu-topology-for-high.html</a></li>
<li>CPU model selection - <a href="http://openstack-in-production.blogspot.fr/2015/08/cpu-model-selection-for-high-throughput.html">http://openstack-in-production.blogspot.fr/2015/08/cpu-model-selection-for-high-throughput.html</a></li>
</ul>
<h3>
References</h3>
<div>
<ul>
<li>Intel article on EPT - https://01.org/blogs/tlcounts/2014/virtualization-advances-performance-efficiency-and-data-protection</li>
<li>Previous studies with KVM and HEP code - https://indico.cern.ch/event/35523/session/28/contribution/246/attachments/705127/968004/HEP_Specific_Benchmarks_of_Virtual_Machines_on_multi-core_CPU_Architectures.pdf</li>
<li>VMWare paper at https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf</li>
</ul>
</div>
<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/01315333079658597119noreply@blogger.com1