August 29, 2009

Data Centre Refurbishment – the Armageddon move

As per my post here – some of you will be aware that my lack of posting has mainly been down to my work on my companies Data Centre. As mentioned before as we have had to do this in situ (where all of the existing server and Communications infrastructure is still in place) which has been especially entertaining considering that as part of the overall works we have had to replace the raised floor.

In order to do this, we have had to complete a series of large moves of the existing kit within the room – the biggest of which happened during the weekend of the 22nd of August.

What made this weekend the hardest and the biggest move is that we had to close down every single server, switch, router, firewall, Call Manager, voice mail server, essentially all 310 servers and everything else that makes my company function from an IT perspective. Additionally we had to remove the existing UPS and Backup Gen Set from the kicking as apart from the moves we were also running in a new power feed to the room (we pretty much took the entire fabric of the room, stopped it from doing what it is supposed to do, and then moved it all around and then plugged it back in again – just for fun ** NOT **).

What also complicated this move was the new power feed. As this was being run from the main building’s electrical switch gear where 1600 AMPs of juice exists on the same bus bars, we had to close down electricity to the entire building as well as the data centre (trust me; this was no mean feat as given that this is a Government building we had to work with the Police and other emergency services).

Bus Bars look like the following:

The picture above does not really do them justice – all you need to remember is if you see them and the power is on then you are too close already! – the juice running through these puppies is incredible.
The Red, Yellow and Blue dots depict the phases (as in 3 phase) – more commonly known these days as L1, L2 and L3 – the Blue cable represents the Neutral; for more information on this (which believe me is surprisingly interesting if you run a data centre – have a look here:
Now when you consider that 2000 people work at this location and it is of proportional size to Microsoft U.K in Reading this is no mean feat, and indeed it is one that if I even considering doing again, I want someone to shoot my ass on the spot.
The communications alone over a period of two weeks letting people know that they cannot access the building at the said weekend, then re-explaining what “Full Power Down” means – “no your fridge will not work”; “yep that’s right your computer and phone won’t work” – and my personal favourite – “no if you come in at the weekend you cannot make a cup of tea and burn toast”), making provision for automated processes to be re-scheduled and then coordinating the close down of the our systems in the correct order was a project within itself.

The move itself went quite well, as indeed did the power work for the new feed, however we ran into a few problems with the existing power that (I had already warned people about months ago which resulted in me and a few of my team whom could still stand having to work 30 hours to get the systems back – but that is a long story (and rant) that does not belong here) – what I would like to post is some of the picture from the move, as it was a good laugh, and indeed the work, commitment and attitude of all my staff continued to make me very, very proud of them.

So in the spirit of the last post I would like to present some of the photos from the move – you can see that in the few weeks since I lasted posted things have moved on quite a bit from the original state of the room – we are currently two weeks ahead of schedule and – dare I say it within budget! – hope you enjoy;

Random Musings 0 Replies to “Data Centre Refurbishment – the Armageddon move”