Simwood Status
All Systems Operational
Wholesale VoIP Operational
London ? Operational
Slough ? Operational
Manchester ? Operational
Fax ? Operational
Virtual Interconnect ? Operational
Customer Interfaces Operational
API ? Operational
Portal ? Operational
Network ? Operational
London Network ? Operational
Slough Network ? Operational
Manchester Network ? Operational
Hosting ? Operational
vCenter London ? Operational
vCenter Slough ? Operational
vCenter Manchester ? Operational
Colo London ? Operational
Colo Slough ? Operational
Colo Manchester ? Operational
Support Operational
Operations Desk ? Operational
Porting Desk Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Apr 26, 2017

No incidents reported today.

Apr 25, 2017

No incidents reported.

Apr 24, 2017

No incidents reported.

Apr 23, 2017
Completed - This has now been completed. Thanks for your patience and understanding.
Apr 23, 18:04 UTC
In progress - This test has commenced as planned.
Apr 21, 23:00 UTC
Scheduled - We have been evaluating new DDoS mitigation partners and are at the stage of trialling one with live traffic to tune / evaluate their detection algorithms. As such we will be announcing all Slough (and anycast) prefixes through them during this window, causing some traffic to reach us this way. This has been selected to be a quiet time on the network. The following should be noted though:

- this only affects people reaching us over IP Transit and to some extent public peering; those who are directly connected and/or privately peer will be unaffected and their traffic will continue to flow directly. Those who are directly on-net in Slough will only be affected in relation to traffic reaching them over IP Transit. [1]

- traffic traversing this route will see slightly reduced MTU for this test which may cause fragmentation issues for those still sending SIP over UDP rather than TLS/TCP. Those affected by this and unwilling/unable to change transport may prefer to send traffic to Manchester for the duration of this test. Changing transport remains our advice. [2]

This is an important further step in ensuring we're able to defend the network against an attack larger than the already substantial on-net provision in-place. We appreciate your understanding.

Naturally, we'll be monitoring throughout and will roll back if any issues are observed.

[1] http://blog.simwood.com/2015/12/surviving-somebody-elses-ddos/
[2] http://blog.simwood.com/2016/03/sip-signalling-tcp-or-udp/
Apr 19, 11:31 UTC
Apr 22, 2017

No incidents reported.

Apr 21, 2017
Completed - This was completed successfully. The issue was identified and we have a plan to overcome it.
Apr 21, 01:32 UTC
Scheduled - As described more fully in the earlier incident report, we will be troubleshooting the legacy Brocade network tonight, and introducing new links to the new Arista-based network.
Apr 20, 10:42 UTC
Apr 20, 2017
Resolved - Apologies for the delay in updating; we have been trying to clarify what happened here.

At broadly 9.09 we began receiving a high volume of internal alerts suggesting connectivity issues between Slough and London and Slough and Manchester. We did not however receive any off-net alerts for any services within the network in any site, only for certain routers. Taken together these suggested an issue cross-network between Slough and London, but one that had not caused backup paths to become active. Shutting down that [active and apparently functional] link caused traffic to flow via Manchester, as we'd normally expect to happen, and re-enabling it some time later restored normality. This was somewhat more convoluted in that the issue only affected two (of 14) routers (one of the legacy Brocade pair in Slough and one of the legacy Brocade pair in Telehouse East), whilst others had correctly changed state. At this time we suspect this to be a Brocade software issue which will be resolved by the path we're already on to replace the legacy Brocade network with the new Arista equipment.

Thankfully our topology means that generally only internal traffic traverses this link and generally speaking for voice traffic, the first hop on our network will be the site that services the call or media. Thus, we did not reroute any calls and from support tickets and calls to our office it seems the affect of this on customers was minimal. It appears internal traffic for billing and services such as CDRs in the portal were directly affected but other than at the moments of state transition, calls were not. Calls to our office were however given these do traverse the network. If you were affected but have not opened a ticket we'd be happy to investigate why.

Moving forwards, we will schedule some emergency maintenance for tonight to troubleshoot this issue further. Unfortunately this does risk repeating it. We will take opportunity to complete the addition of new links between new equipment. This is Arista to Arista and connects Slough LD4 directly to both Telehouse North and Volta. Depending on the outcome of this the legacy Brocade to Brocade Slough LD4 to Telehouse East link may be retired. We're due to blog about these network changes next week anyway by way of explanation for all the maintenance windows lately.

Apologies for any inconvenience caused.
Apr 20, 10:21 UTC
Monitoring - This has been rectified and is being observed/investigated further.
Apr 20, 08:34 UTC
Investigating - We are investigating an incident and will advise more in due course.
Apr 20, 08:17 UTC
Apr 19, 2017

No incidents reported.

Apr 18, 2017
Completed - Completion of this work will be rescheduled to a future window.
Apr 18, 23:39 UTC
Update - We are fully patched and configured but require third party patching to be completed. This has been requested and is awaited but if not executed inside the maintenance window, links will be shutdown and completion of this work rescheduled.
Apr 18, 23:22 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 18, 23:00 UTC
Scheduled - We will be enabling two additional links on the network, further improving capacity and resiliency. Physical patching of the disabled links in Slough will take place before the start of the window (late afternoon) and is not expected to have any impact whatsoever. The links themselves will be enabled during this window which will cause traffic to reroute.
Apr 13, 12:21 UTC
Apr 17, 2017

No incidents reported.

Apr 16, 2017

No incidents reported.

Apr 15, 2017

No incidents reported.

Apr 14, 2017

No incidents reported.

Apr 13, 2017
Completed - This maintenance has completed. Unfortunately there were some unintended network stability consequences in other sites for which we're sorry.
Apr 13, 04:31 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 12, 20:09 UTC
Scheduled - We will be inserting new Arista equipment to the network but this should be less intrusive work than previously as it should not impact the existing network and inter-site links. We will however migrating our Telia transit to this new equipment which will cause re-routing of a minority of off-net traffic for a period. Finally, a limited amount of equipment will be re-patched into this which may affect service from Slough of our anycast Portal (Manchester nodes will however pick up the load). This is the final physical phase of our network upgrade.
Apr 9, 10:14 UTC