Category: They did WHAT?!

So I heard you want redundancy AND space? Let’s get you on a RAID… 0?!

November 25, 2014 /

That’s right! Over the last week I had been communicating with a client that uses us to host one of their clients and were looking at possibly doing an emergency restore of their data to our platform. Great, right?

The reason behind them needing an emergency deployment in our systems was found out today. See, I knew that there was a catastrophic drive failure, which resulted in the loss of about a week or so of data (since that was the last time they did off server backups!). What I did not know, and found out today, that this server was not one they setup (thank god), but instead inherited from the previous IT. The previous group had been given the specs by GE for their CPS product, every time we get a new client we have to go through the hoops required to spec out a clients environment, even though we have ours tuned perfectly for us.

What bugs me is that in this calculator they give you, a 10 doc practice can be advised to have a database server with 14 spindles as a minimum.. Wait.. 14?! Yes, that’s right, 14 as a minimum, and that’s not all, they have to be SAS or SCSI drives, or have so many spindles for SATA that your paying up the arse to host your product.

I get it though, the more drives the more redundancy and the better read and write times you have, for a product that has been pieced together similar to Frankenstein’s creation, you have to set the bar high to make up for any programming issues (and don’t get me started on them).

This leads us back to why i’m writing this post.. This client’s former IT, with the knowledge of the recommended setup, consulted, procured and setup a solution that was 100% vm, but did so in a way that just makes me wonder how people are in IT to begin with.

The kicker? They somehow managed to setup this virutal host with a RAID 0.. Not a RAID 5 which was expected/thought had been done, but a raid 0.. I mean, I can see a RAID 0 for two drives when you want tempdb to be as fast as possible and with as much possible space. But hell, that’s tempdb.. if it takes a dive, it’s not the end of the world, (necessarily..) instead a system was setup, that served VMs (OS unknown), ran 24/7 and had monitoring, but your setup the drives as a giant raid 0.. way to go.. just.. wow.

I’ve played the various thoughts through my head after hearing about this, trying to play out how a Virtual Host with MISSION critical items on it, could be deployed as such a setup, with ZERO redundancies, it just astounds me. Now the client is looking at a week or two of downtime, and thousands of dollars to go to a data recovery company because of some incompetent IT guy (thanks to him/her for giving those of us with half a brain a bad name..).

Moral of this story guys? Don’t use a raid one when you have missing critical software on it, it’s stupid, and it’s dangerous..

What NOT to do on a Network… *shakes head*

November 11, 2013 /

So, I’ve been in Kansas City to turn up a nice new client for our Cloud Hosting, since their internet had not been kind to letting me on to check things out, we got onsite and had a not-so warm welcome to their IT infrastructure..

After arriving and checking out the machines and networking equipment, we found the following:

1. The domain, was 2003 based domain, that was something like: r512A3.com (I could only shake my head at this)
2. Workstations had no solid naming convention, after going to ~10 workstations, I did find a pattern.. they were all windows xp default naming convention -_- (the random letter and numbers, wtf!)
3. There were a total of about 35-40 PCs and Wyse terminals, but they had 3, 24 port racked switches, and 6 hubs.. yes.. hubs.. so the reason they kept loosing connectivity or had random dropping of connections, yup that daisy chained hub was the issue..
4. And the straw that broke the camel’s back per-say.. EVERY machine was a static IP config… yeah *everyone* oh, and who wants to guess why..? well.. it’s easy.. it’s because the old IT never setup the DHCP role on the DC.. awesome..

So now that your head has to hurt as much as mine has through the last day.. my rant is over.. I understand that some things are a preference of the IT that set it up (using a .com TLD instead of .local for a local domain [I prefer .local because of what it is..]), but gosh how the HELL can someone eff up a domain so badly..

After setting up DHCP on our new firewall, changing all of the local machines to DHCP and setting up all of the printers in the firewall’s DHCP reservation, we were on track for a good morning.

Fast forward to this morning, we got in and luckily the work we did last night helped immensely, there was some printing hiccups as we made sure to iron out all of the details for our cloud platform. Combine the printers with getting an emergency call to a local cable guy, we were able to do 8 new runs (btw, this place is 11k square feet!), to allow us to remove the Hubs, and hook up our three new wireless access points, score!

At the end of the day, we found ourselfs with a solid network, a wireless network that was ready for the docs to use and things looking good.

Now that I have my ranting out, i’m off for the night.. for those that read these, thanks, for those that don’t, you might be missing out.

Peace
-B

Long time.. How do you like your fiber… cut?

October 22, 2013 /

So it’s been a while.. I was really getting in to doing a post once a day, then life/work got in the way and totally blew the wind out of my sails..

Anyway.. There’s been a ton that has happened since my last post.. The most interesting thing I ran into over the last few weeks was a fiber cut for a major ISP..

For those of you that don’t know how networking is setup, there are different ways to get your data traffic around, Copper (Ethernet, Coax), Fiber and Air (WiFi, P2P Long distance Directional Wireless). A week or so back we had two clients call in and report that their internet was down.. Well, the good news was we knew their internet was down due to our in-house monitoring system (gosh is it awesome..).

So.. after an hour or two of nothing coming back up, I gave a call to their provider, in this case, was Vinakom.. While I normally have no issues with a company selling T1’s and charging 300+ a month for them, what I DO have an issue is, when you have clients in Evanston, and Elgin IL and both of their connectivity to the world goes down.. Wait.. the whole world?!

Yes. Locations >35 Miles away from each other lost connectivity, they were 100% down.. When I called the ISP was greeted promptly, but was told that one of their Fiber lines were cut, that it was minimal and would only effect a few clients.. hah wait, only a few!? While the woman answering the phone was nice, I was NOT impressed. How can you as an ISP have a single fiber cut but downplay the effects on their clients! Ugh.

Needless to say, the outage lasted 7 hours, it wasn’t until 4:30pm that the connectivity came back…

Minor Fiber cut my butt.. To this day I still believe Vinakom is only peering with one provider, this is not only stupid but an idiotic thing to do.. especially when it is your lifeblood.

Either that.. or the pipe that they used to go to the ISP hotel on 350 Cermak.. now THAT would be hilarious..

Oh well.. alls well that ends well.. I would have advised my clients to jump ship and to use the outage as a way to get out of their contract.. but in both situations they can only get ADSL or a T1.. bummer.

Well that’s all for now, i’ll make it more of a point to get on here and do more updates.. The business of IT changes all the time, and so does the amount of STUFF that we see.

Be back soon.

-b

iPhones.. more reason to hate them..

September 19, 2013 /

So for those that know me, they know that I despise apple products. While I respect their creative usefulness, I despise the marketing and the general operation of apple, they just suck.

Today proved to be another straw in my hat for the fight against apple.

As everyone should be aware (in one shape or form), Apple released their new iOS 7 yesterday, woo…. Well, I took a call this morning, a client of ours was reporting intermittent data connectivity issues. So the tech I am, I checked their connection and found extreme packet loss at the Comcast node one hop from their modem. I explained to the client that it looked like comcast was having issues and that they would need to be contacted.

She explained that she had already talked to them, and to her dismay had been told that it was something coming from within their network as the modem looked fine.. Even though it clearly wasn’t. Things like that enrage me, a tech just telling an end user that its their network and they had no issues -_-

So I called Comcast, found that they were having issues at the node above the modem, although the tech did not see any issues with the modem. While I was talking with this tech, I was told that they already had someone dispatched, that they would be at the location in the afternoon. I called back the client and told them that we would just have to see what the Tech saw.

Fast forward to 4:45, I got a call from this same client, she said the tech was onsite and wanted to talk to me. Talking with the tech, he had been doing tests and found that when he unplugged the uplink to our firewall, the connectivity and response times were back to normal levels, the hell right?

Well, since the issue was an issue within the network, I dismissed the tech and started working with the end user. One of the things I was taught with Juniper’s is that they have a superb logging system, at least from my point of view..

I turned on logging on the outside port, and to my surprise, I watched as large data sets ran across going to microsoft and apple IPs. After figuring out that the Microsoft IPs were for checking of windows updates, I narrowed in on the apple ips. The rDNS for the apple ips resolved to a cdn that was dedicated to an iOS 7 distribution, so I pushed a little further, I asked the end user what types of phones they had in the office and she said everyone had an iPhone. Great. I found the culprit, now just to figure out a way to tell the end user that their phones were killing their network.

Lucky for me, the end user I work with at this specific location is great to work with, she is very relaxed even when there are real legit issues. So I explained to her, that it appeared the iPhones were downloading updates, which were causing the network traffic to go crazy. She asked around and checked a few phones, she said that everyone was connected to their wifi, and that there were some that were even in the process of downloading the updates to the system and apps!

So that’s where we end, the end user said she was going to get all of the phones off the network and then we would see how things settle out in the next day or so.

tl;dr iPhone updates suck, they didn’t do rolling updates or anything that would allow them to stagger their updates to help networks, they just pushed it out and expected everyone to understand.. Damn iPhones.

-B