That’s right! Over the last week I had been communicating with a client that uses us to host one of their clients and were looking at possibly doing an emergency restore of their data to our platform. Great, right?
The reason behind them needing an emergency deployment in our systems was found out today. See, I knew that there was a catastrophic drive failure, which resulted in the loss of about a week or so of data (since that was the last time they did off server backups!). What I did not know, and found out today, that this server was not one they setup (thank god), but instead inherited from the previous IT. The previous group had been given the specs by GE for their CPS product, every time we get a new client we have to go through the hoops required to spec out a clients environment, even though we have ours tuned perfectly for us.
What bugs me is that in this calculator they give you, a 10 doc practice can be advised to have a database server with 14 spindles as a minimum.. Wait.. 14?! Yes, that’s right, 14 as a minimum, and that’s not all, they have to be SAS or SCSI drives, or have so many spindles for SATA that your paying up the arse to host your product.
I get it though, the more drives the more redundancy and the better read and write times you have, for a product that has been pieced together similar to Frankenstein’s creation, you have to set the bar high to make up for any programming issues (and don’t get me started on them).
This leads us back to why i’m writing this post.. This client’s former IT, with the knowledge of the recommended setup, consulted, procured and setup a solution that was 100% vm, but did so in a way that just makes me wonder how people are in IT to begin with.
The kicker? They somehow managed to setup this virutal host with a RAID 0.. Not a RAID 5 which was expected/thought had been done, but a raid 0.. I mean, I can see a RAID 0 for two drives when you want tempdb to be as fast as possible and with as much possible space. But hell, that’s tempdb.. if it takes a dive, it’s not the end of the world, (necessarily..) instead a system was setup, that served VMs (OS unknown), ran 24/7 and had monitoring, but your setup the drives as a giant raid 0.. way to go.. just.. wow.
I’ve played the various thoughts through my head after hearing about this, trying to play out how a Virtual Host with MISSION critical items on it, could be deployed as such a setup, with ZERO redundancies, it just astounds me. Now the client is looking at a week or two of downtime, and thousands of dollars to go to a data recovery company because of some incompetent IT guy (thanks to him/her for giving those of us with half a brain a bad name..).
Moral of this story guys? Don’t use a raid one when you have missing critical software on it, it’s stupid, and it’s dangerous..