Service Status

Service Status

Platform Incident - ASSURED/ELEVATED - Enterprise Cloud Compute - Region 1, 2, 7 and 8 Thursday 8th February 2018 10:41:00


We are experiencing a number of problems with the vCenter control plane in these regions due to bugs introduced by a recent upgrade.

This problem can affect the following operations: VM creation, adding new disks to VM, Edge Redeployment.

We have engaged VMware support to work through to a resolution as a matter of urgency.

We will be undertaking recovery work with VMware which may involve the stopping and starting of services. This will briefly cause loss of management control to customer VMs in these regions.

Snapshot protection services have been impacted as a symptom of the control plane problems.

Incident resolved.

UPDATE: 24 Feb We have completed emergency patching across Regions: 1 (Zones: AE1, AF1, AF2, AF3) & 2 (Zones: AE1, AC1, AC2, AC3) and the Control Plane and Snapshot Protection services have been re-enabled. We are hopeful that VM creation, adding new disks to a VM and Edge Redeployment tasks should now complete successfully. We will continue to monitor the state of these tasks over the coming week and will update this incident with more information regarding the effectiveness of the applied patches.

UPDATE: 24 Feb We have completed emergency patching across Regions: 1 (Zones: AE1, AF1, AF2, AF3) & 2 (Zones: AE1, AC1, AC2, AC3) and the Control Plane and Snapshot Protection services have been re-enabled. We are hopeful that VM creation, adding new disks to a VM and Edge Redeployment tasks should now complete successfully. We will continue to monitor the state of these tasks over the coming week and will update this incident with more information regarding the effectiveness of the applied patches.

UPDATE: 23 Feb The aforementioned emergency patch from VMware will be applied to Regions: 1 (Zones: AE1, AF1, AF2, AF3) & 2 (Zones: AE1, AC1, AC2, AC3) between 08:00 on 24/02 - 20:00 on 25/02. During this period access to the Control Plane will be disabled as will Snapshot Protection to allow the patches to be applied with minimal disruption. These will be re-enabled once patching work is completed.

UPDATE: 20 Feb
We completed our testing of VMware’s emergency patch and have implemented into Region 7/8 Zone EE1. Subject to further testing, we may role out to regions EC1 and EF1 later this week. We will continue our consolidation work, which will incur vMotions. Furthermore, there may be requirements to restart the Control Plane across Regions 1,2,7 & 8 to allow for the implementation of further fixes and recommendations from our vendor.

We anticipate further patches from VMware later this week

UPDATE: 16 Feb

VMware have released an emergency patch to address the identified bug. We are currently testing and will be implementing the patch, initially in Region 7/8 Zone 10/11 EE1. As stated earlier, the Control Plane across Regions 1,2,7 & 8 will undergo emergency restarts as we implement fixes and recommendations from our vendor. Our consolidation work involving vMotions, continues in the background.

UPDATE: 14 Feb As part of the remedial actions to help address the problem around VM creation, Adding new disks to VM and Edge Redeployment. We are consolidating some of the backend storage repositories and compute clusters to improve efficiencies across the platform. During this consolidation process, your VM may undergo a vMotion. There is no expected impact, however on some occasions high IO VMs may experience negligible latency.

UPDATE: 09 Feb.
We have been working with our vendor to resolve the current problems. VMware have confirmed that other suppliers have experienced similar problems and are escalating internally to identify the root cause. We have applied recommended fixes with limited success. We have now isolated the impact to the following zones (2-AC3, 2-AC2, 11-EE1, 10-EF1). If you experience any of the symptoms described below, please retry. If that fails, contact the UKCloud support desk for assistance.