Incident Details

Resolved [02/07/2021 13:44]

Whilst investigating a degraded performance issue on a dark fibre at our LD8 PoP, a third party engineer inadvertently disconnected another dark fibre that connects LD8 to a third location. This subsequently resulted in LD8 becoming isolated from the rest of the network for a short period, between 00:06:02 and 00:09:56.

As previously reported, during this time leased line circuits terminating at LD8 would have experienced a loss of connectivity. Broadband circuits were impacted further due to a large number of subscriber sessions that were terminating at LD8 disconnecting.

Whilst the majority of the affected broadband subscribers regained a session at another PoP relatively quickly, others whose sessions were steered to a particular aggregation router on the network failed to start. Our engineers investigated and discovered that the router was experiencing a fault condition and took it out of service. At this point the vast majority of remaining subscribers re-gained their sessions.

Apologies for the disruption this may have caused.

Update [02/07/2021 02:29]

We are now seeing the vast majority of broadband subscribers back in session. We'll provide further RFO details later today once everything is understood. Apologies for the disruption caused.

Identified [02/07/2021 02:03]

We are aware of an issue this evening impacting leased line and broadband connectivity. The issue has been caused by a third party dark fibre provider. Leased line circuits impacted may have experienced disruption of up to 5 minutes. Broadband circuits may have had a longer window of disruption and we are actively working to recover lost sessions.