We've now re-enabled Level3 but will continue to monitor.
Posted Jun 12, 2015 - 12:51 UTC
Monitoring
Untrusted sources are claiming this is resolved but we are leaving our Level3 ports disabled for the time being.
We have also made a number of routing changes amongst peers in an attempt to circumvent congestion on their network resulting from traffic shifts.
We continue to monitor the situation.
Posted Jun 12, 2015 - 11:43 UTC
Update
By way of further information we understand the issue may have originated with Telekom Malaysia who leaked routes to Global Crossing (AS3549) which were trusted by Level3 (AS3356), the two now being the same company and combined making the largest tier1 network on the planet. This resulted in Level3 tripping max prefix limits with their peers, who are by definition other tier-1 operators, and their sessions being disabled. This has resulted in large swathes of the Internet being unreachable from, to, and over Level3 and increased traffic over other routes as a result. This has not been confirmed by Level3 but is our understanding of events.
As mentioned, our ports to Level3 are disabled so the Simwood network is reachable over other routes. End-users behind ISPs who continue to route via Level3 will however continue to see issues reaching us and other locations until either their ISP disables Level3 or Level3 restore stability to their network.
Posted Jun 12, 2015 - 10:43 UTC
Identified
We have confirmed that there is a major issue in the Level3 network.
We have shut down all connectivity between ourselves and Level3 so all traffic to Simwood will transit over unaffected networks. Customers using ISPs reliant on Level3 will continue to have connectivity issues and should speak to their supplier directly.
Posted Jun 12, 2015 - 10:20 UTC
Investigating
We have had reports of connectivity issues from customers using Level3 - this appears to be an issue outside of our network.