At approximately 9:15 AM the college website was inadvertently taken offline due to a bug triggered through the management dashboard of our provider.
A routine update to our website app where the application runtime is reloaded to apply changes triggered a bug which caused a new load balancer to be provisioned but was not initialised correctly, this caused all request to our app to timeout. Because of this, the app was put in state where it could not be recovered through normal actions in the dashboard. After initial attempts to remedy the situation ourselves we alerted our provider at 9:17 am to the issue, we received acknowledgement of the report at 9:28 am.
Our provider quickly responded and began investigating the issue. At 9:52 am the issue was identified and a fix implemented. At around 9:55 am, the application that serves our website was restored after resolving the issue with the load balancer configuration that had led to the problem.
Normally, dashboard actions should not trigger such events and this was acknowledged as a bug within our providers platform. The issue is highly unusual and rare. However as a future policy, we will look to defer any such changes that reload the app runtime unless absolutely critical going forward. Normally the action performed should not cause issues.