Announcements
You are here: Support > Announcements > Down Time

29/12/2008 23:40

Down Time

Sorry for the recent and on-going down time, here is a over view of the problem from the Data Centre:

After what can only be described as the maintenance window from hell, I felt it only fair to explain why there was such a delay in getting everything back online today.  To keep things simple I'll break each individual issue that lead to today’s delay into its own paragraph.

1.) Spanning Tree Issues

During today’s maintenance window we re-installed our primary core switch as we had a temporary 'fill-in' in place at KSP whilst our HP 2824 from iPH was upgraded to the latest firmware.  Unfortunately the re-installation of this switch did not go according to plan as every time we plugged rack uplinks into the core switch we disappeared off of the face of the internet.  After about an hour of trouble shooting with our upstream (who luckily we had asked to attend the Data Centre with us) we found the issue was due to a spanning tree conflict.  We take two feeds from our upstream (one for SW transit and one for our BF backhaul to London).  Despite these two feeds being provided over two separate VLAN tags and thus virtually independent from each other, for some reason the switches both end detected a possible loop and were automatically blocking all traffic to and from our upstream making it look to us as if there was a physical cabling loop!

2.) Window Duration

This one is completely our fault, we totally underestimated how long the window would take and in hind sight, if we had of worked out our timings better then the window would have been scheduled for the early hours of the morning to ensure the least disruption was caused to customers.

To summarise, several things completely out of our control played a large part in the delay today but we will most definitely be paying more attention to our window planning in future to ensure that where possible, we've allocated enough time to cover for any mishaps!

Lastly, I would like to thank everyone for their patience and understanding today.

As soon as we have an update we will let you know :D May I wish all our customers a very prosperous 2009.

<< Back

View RSS Feed

Powered by WHMCompleteSolution