Archive for June, 2008

Wait… Backup, I Missed Something!
Posted by Justin Scott on June 4th 2008

I’ve been around computers all my life (OK, since 1977 but that’s almost all my life) and was lucky to get my first computer in 1983.

Over the summer of 1984, I was deeply embroiled in (up to that point) the largest programming project of my life, coding Z80 ASM on my trusty CP/M computer when I encountered the most dreaded of all BDOS errors, “BDOS ERROR ON B: BAD SECTOR”

In its most mild form, this cryptic message simply means “copy this data to another disk before this one fails.” However, in this specific instance, it represented the most severe case… “this disk is toast, kaputt, finito, your data is GONE!!!”

Via the School of Hard Knocks, I learned the value of keeping proper backups that day.
If you’ve been in this game for longer than about 10 milliseconds, it’s probable that you’ve experienced data loss in one form or another. Over the years, I’ve seen just about every kind of data loss imaginable, from the 1980’s accountant who tacked her data floppy to the filing cabinet with a magnet so she wouldn’t misplace it– all the way to enterprise/mainframe class SAN equipment that pulverizes terabytes of critical data in less than a heartbeat due to operator error on the part of a contractor.

I’ve consulted with thousands of individuals and companies about their backup implementations and strategies, and am no longer surprised by administrators who believe they have a foolproof backup utilizing a secondary hard disk in their systems. I have witnessed disk controller failures which corrupt the contents of all attached disk drives, operator error and/or forgetfulness that leave gaping holes in so-called backup strategies and other random disasters. On the other side of the coin, I have personally experienced tragic media failure from “traditional backups” utilizing removable media such as tapes and/or CD/DVD/etc.

Your data is your life. I’ve waited up until this point to mention this, because it should be painfully obvious to every administrator, but in my experience the mentality is along the lines of “My data exists, therefore it is safe.” What happens when your data ceases to exist, and you become aware of the flaws in your backup plan? I’ll tell you – you go bankrupt, you go out of business, you get sued, you lose your job, you go homeless, and so-on. Sure, maybe those things won’t happen to you, but is your livelihood worth the gamble?

“But Justin… my data is safe because it’s stored on a RAID mirror!” I disagree. Your data is AVAILABLE, your data is FAULT TOLERANT, but it is not SAFE. RAID controllers fail. Disaster happens. Disgruntled or improperly trained personnel type ‘rm –rf /’ or accidentally select the wrong physical device when working with the Disk Manager in Windows. Mistakes happen. The unforeseeable, unavoidable, unthinkable happens.

Safe data is geographically diverse data. Safe data is up-to-date data. Safe data is readily retrievable data. Safe data is more than a single point-in-time instance.

Unsafe data is “all your eggs in one basket.” Unsafe data is “I’ll get around to doing that backup tomorrow.” Unsafe data is “I stored the backups at my house which is also underwater now.” Unsafe data is “I only have yesterday’s backup and last week’s backup, and this data disappeared two days ago.”

SoftLayer’s customers are privileged to have the option to build a truly safe data backup strategy by employing the Evault option on StorageLayer. This solution provides instantaneous off-site backups and efficiently utilizes tight compression and block-level delta technologies, is fully automated, has an extremely flexible retention policy system permitting multiple tiers of recovery points-in-time, is always online via our very sophisticated private network for speedy recovery, and most importantly—is incredibly economical for the value it provides. To really pour on the industry-speak acronym soup, it gives the customer the tools for their BCP to provide a DR scenario with the fastest RTO with the best RPO that any CAB would approve because of its obvious TCR (Total Cost of Recovery). Ok, so I made that last one up… but if you don’t recover from data loss, what does it cost you?

On my personal server, I utilize this offering to protect more than 22 GB of data. It backs up my entire server daily, keeping no less than seven daily copies representing at least one week of data. It backs up my databases hourly, keeping no less than 72 hourly copies representing at least three days of data. It does all this seamlessly, in the background, and emails me when it is successful or if there is an issue.

Most importantly, it keeps my data safe in Seattle, while my server is located in Dallas. Alternatively, if my server were located in Seattle, I could choose for my data to be stored in Dallas or our new Washington DC facility. Here’s the kicker, though. It provides me the ability to have this level of protection, with all the bells and whistles mentioned above, without overstepping the boundary of my 10 GB service. That’s right, I have 72 copies of my database and 7 copies of my server, of which the original data totals in excess of 22 GB, stored within 10 GB on the backup server.

That’s more than sufficient for my needs, but I could retain weekly data or monthly data without significant increase in storage requirements, due to the nature of my dataset.
This service costs a mere $20/mo, or $240/yr. How much would you expect to pay to be able to sleep at night, knowing your data is safe?

Are you missing something? Wait… Backup!

 
Lights Out!
Posted by William Francis on June 2nd 2008

A couple weeks ago, I made a quick stop by my friendly neighborhood drug store to pick up some film for my camera. When I came out of the store, I hopped in my car, turned the key, and then… BOOM! That’s right—there was an explosion. Nothing earth shattering, but alarming none-the-less. The explosion was certainly loud enough to turn a few heads. And it gave me a bit of a scare as there was a moment where I found myself wondering if my ex wife had finally saved up enough money to have me taken out.

After giving the smoke a few minutes to clear out, and my heart a few beats to find its way out of my throat and back into my chest, I got out and looked under the hood. Not because I know anything about what makes a car tick, but because looking under hoods is what guys do. Especially when people are watching. In fact, another guy nearby came over and looked under the hood too. And as soon as he opened his mouth I realized that like me, he was only looking under the hood because that is what guys do in these situations.

“I don’t think your battery is supposed to have a big crack down the middle like that,” he said to me.

“Me neither,” I mumbled. I suppressed the urge to ask Mr. Obvious where he went to automotive school.

“You try it again?” he asked me.

For a second I thought he was joking. Then I realized he wasn’t and more importantly that I didn’t have any better ideas. So I hunted around the engine block until I found where the top of the battery case had been propelled, just sort of laid it over the smoking remnants of the battery, then slid back into the car and tried the key. It didn’t start. When I turned the key all the lights came on, all the dials and gauges swung wildly from side to side, and then everything went dark. Lights out. I tried again. But this time there weren’t even any lights. My sporty Mazda 6 might as well have turned into a pumpkin for all the good it was going to be getting me home. So much for zoom, zoom, zoom.

“You might need to call for a tow,” said Nostradamus now standing by my car window with his hands in his pockets.

“Thanks again,” I said unenthusiastically, “I got it from here.” I hoped he’d take the hint.

He did.

I waited till I saw him drive off then tried the key one more time. Nothing. So I broke down and called the towing service. Rather than calling a friend or a taxi, I opted to walk the two miles home from the drug store. During which I had some time to think. It occurred to me that the idea I might start up my car and drive it to the nearest shop for diagnostics after it just got done blowing up in my face was pretty outlandish. And yet, that is exactly what we offer with the servers we sell at SoftLayer.

Lights Out Management (LOM) or Out Of Band Management (OOB) as it is sometimes called is a feature we include with all of our servers at SoftLayer. If you’re a current customer, you have probably noticed the “management ip address” noted for your servers in our portal. That’s exactly what I’m referring to. And while LOM is the stuff of science fiction in automobiles, in our world class servers it’s a reality. That’s right, with our OOB offerings you can:

  • Cycle the power on a server even if the operating system has crashed, locked, or otherwise blown up.
  • Start up a dead server with the push of a button.
  • Get critical readings of system health indicators like processor temperature and fan speeds, regardless of what operating system or software you have installed on that server.
  • Manipulate system BIOS and perform diagnostics remotely with full video, keyboard, and mouse support.
  • And a whole slew of other things that will make your life much much easier.

In essence, SoftLayer’s OOB management features are the next best thing to driving to one of our data centers and plugging a keyboard into your server. Maybe even better, since you don’t have to fight the traffic. It’s the sort of thing a system administrator dreams of. The sort of thing that sets SoftLayer apart from the myriad of other hosting companies out there.

As for my car, two days and two hundred dollars later it was back in my driveway. Apparently the mechanic who worked on my vehicle went to the same school of engineering as the Good Samaritan who provided me so much insight two days prior. The diagnosis, according to the invoice, was: cracked battery. Then just to give me a warm and fuzzy written in big block letters at the bottom of the page was:

IF THIS HAPPENS AGAIN BRING IT BACK IN

I can’t wait till the automotive industry catches up to SoftLayer.

 










 
 
Copyright © SoftLayer Technologies, All Rights Reserved.
Close
E-mail It
Socialized through Gregarious 42