Apologies for the delay in updating; we have been trying to clarify what happened here.
At broadly 9.09 we began receiving a high volume of internal alerts suggesting connectivity issues between Slough and London and Slough and Manchester. We did not however receive any off-net alerts for any services within the network in any site, only for certain routers. Taken together these suggested an issue cross-network between Slough and London, but one that had not caused backup paths to become active. Shutting down that [active and apparently functional] link caused traffic to flow via Manchester, as we'd normally expect to happen, and re-enabling it some time later restored normality. This was somewhat more convoluted in that the issue only affected two (of 14) routers (one of the legacy Brocade pair in Slough and one of the legacy Brocade pair in Telehouse East), whilst others had correctly changed state. At this time we suspect this to be a Brocade software issue which will be resolved by the path we're already on to replace the legacy Brocade network with the new Arista equipment.
Thankfully our topology means that generally only internal traffic traverses this link and generally speaking for voice traffic, the first hop on our network will be the site that services the call or media. Thus, we did not reroute any calls and from support tickets and calls to our office it seems the affect of this on customers was minimal. It appears internal traffic for billing and services such as CDRs in the portal were directly affected but other than at the moments of state transition, calls were not. Calls to our office were however given these do traverse the network. If you were affected but have not opened a ticket we'd be happy to investigate why.
Moving forwards, we will schedule some emergency maintenance for tonight to troubleshoot this issue further. Unfortunately this does risk repeating it. We will take opportunity to complete the addition of new links between new equipment. This is Arista to Arista and connects Slough LD4 directly to both Telehouse North and Volta. Depending on the outcome of this the legacy Brocade to Brocade Slough LD4 to Telehouse East link may be retired. We're due to blog about these network changes next week anyway by way of explanation for all the maintenance windows lately.
Apologies for any inconvenience caused.
Posted about 1 year ago. Apr 20, 2017 - 10:21 UTC
This has been rectified and is being observed/investigated further.
Posted about 1 year ago. Apr 20, 2017 - 08:34 UTC
We are investigating an incident and will advise more in due course.