A couple weeks ago, I made a quick stop by my friendly neighborhood drug store to pick up some film for my camera. When I came out of the store, I hopped in my car, turned the key, and then… BOOM! That’s right—there was an explosion. Nothing earth shattering, but alarming none-the-less. The explosion was certainly loud enough to turn a few heads. And it gave me a bit of a scare as there was a moment where I found myself wondering if my ex wife had finally saved up enough money to have me taken out.

After giving the smoke a few minutes to clear out, and my heart a few beats to find its way out of my throat and back into my chest, I got out and looked under the hood. Not because I know anything about what makes a car tick, but because looking under hoods is what guys do. Especially when people are watching. In fact, another guy nearby came over and looked under the hood too. And as soon as he opened his mouth I realized that like me, he was only looking under the hood because that is what guys do in these situations.

“I don’t think your battery is supposed to have a big crack down the middle like that,” he said to me.

“Me neither,” I mumbled. I suppressed the urge to ask Mr. Obvious where he went to automotive school.

“You try it again?” he asked me.

For a second I thought he was joking. Then I realized he wasn’t and more importantly that I didn’t have any better ideas. So I hunted around the engine block until I found where the top of the battery case had been propelled, just sort of laid it over the smoking remnants of the battery, then slid back into the car and tried the key. It didn’t start. When I turned the key all the lights came on, all the dials and gauges swung wildly from side to side, and then everything went dark. Lights out. I tried again. But this time there weren’t even any lights. My sporty Mazda 6 might as well have turned into a pumpkin for all the good it was going to be getting me home. So much for zoom, zoom, zoom.

“You might need to call for a tow,” said Nostradamus now standing by my car window with his hands in his pockets.

“Thanks again,” I said unenthusiastically, “I got it from here.” I hoped he’d take the hint.

He did.

I waited till I saw him drive off then tried the key one more time. Nothing. So I broke down and called the towing service. Rather than calling a friend or a taxi, I opted to walk the two miles home from the drug store. During which I had some time to think. It occurred to me that the idea I might start up my car and drive it to the nearest shop for diagnostics after it just got done blowing up in my face was pretty outlandish. And yet, that is exactly what we offer with the servers we sell at SoftLayer.

Lights Out Management (LOM) or Out Of Band Management (OOB) as it is sometimes called is a feature we include with all of our servers at SoftLayer. If you’re a current customer, you have probably noticed the “management ip address” noted for your servers in our portal. That’s exactly what I’m referring to. And while LOM is the stuff of science fiction in automobiles, in our world class servers it’s a reality. That’s right, with our OOB offerings you can:

• Cycle the power on a server even if the operating system has crashed, locked, or otherwise blown up.
• Start up a dead server with the push of a button.
• Get critical readings of system health indicators like processor temperature and fan speeds, regardless of what operating system or software you have installed on that server.
• Manipulate system BIOS and perform diagnostics remotely with full video, keyboard, and mouse support.
• And a whole slew of other things that will make your life much much easier.

In essence, SoftLayer’s OOB management features are the next best thing to driving to one of our data centers and plugging a keyboard into your server. Maybe even better, since you don’t have to fight the traffic. It’s the sort of thing a system administrator dreams of. The sort of thing that sets SoftLayer apart from the myriad of other hosting companies out there.

As for my car, two days and two hundred dollars later it was back in my driveway. Apparently the mechanic who worked on my vehicle went to the same school of engineering as the Good Samaritan who provided me so much insight two days prior. The diagnosis, according to the invoice, was: cracked battery. Then just to give me a warm and fuzzy written in big block letters at the bottom of the page was:

IF THIS HAPPENS AGAIN BRING IT BACK IN

I can’t wait till the automotive industry catches up to SoftLayer.

Through the virtue of me having young parents, at age nine my own son Taylor has the fortune of not just having grandparents, but great grandparents alive and well. On my mother’s side in particular, I have a grandfather (after who I am named), who is still quite the traveler at age 72. While he lives in Ohio he frequently “pops in” on my son and me. Despite his inability to call and let me know he is coming or how long he will stay, once I get over the initial shock of discovering he is waiting in the driveway for me to come home from work and welcome him into my home, we usually do have a nice visit. (Though he has yet to convince me to give him his own garage door opener despite asking on more than one occasion!)

My son especially likes having his great-grandfather around. My grandfather, as I am sure do most grandfathers, has seventy plus years worth of stories and opinions and riddles he has collected over a remarkably varied life. And if there is one thing that my grandfather is not, it is shy; so my son finds his great grandpa immensely entertaining–as did I at age nine. (Although between you and me I really thought by the time I was nine I had stopped falling for the old pull-my-finger trick that still sends my grandfather and my son into fits of laughter visit after visit.)

The last time my grandfather came out to visit, Taylor had a lot of homework. So after dinner my grandfather settled onto the couch to watch wrestling, (pronounced WRAST’lin), and Taylor and I went about trying to do his homework. The assignment was geared at reinforcing multiplication tables. Something my son struggled with for a bit. So we were working on it for a while. Long enough that my grandfather decided either the school was passing out too much homework, or I was explaining it wrong. So like any concerned great grandparent would do my grandfather clicked off the TV, walked into the kitchen, and pulled up a chair intent on showing us the error of our ways.

Grandpa asked Taylor to explain the assignment–which my son did. Without warning Grandpa then plucked the page and pencil from my son’s hands and proceeded to stare over a multiplication problem for some time. The page stared back at him.

128 x 69 =

Taylor and I watched with growing fascination as grandpa proceeded to scribble nearly as many figures on the page as there were problems. At last he grunted and wrote his answer.

9088

Now I am by no means a mathematical giant, but something seemed a bit peculiar about his answer. So I did a quick computation and came up with 8,832. And while I was still trying to politely figure out how to tell my grandpa “thanks but no thanks”, my son didn’t show any such discretion.

“That’s wrong Great Grandpa!” he exclaimed.

My grandfather took the page back, made some more of his calculations in the margin, then looked up in all seriousness and said to my son:

“Taylor, you are old enough to know the truth.”

I have to tell you at this moment I was pretty shocked. While I was not sure what great personal revelation my grandfather intended to make, I was sure it was to be a difficult one. Every father and grandfather and great grandfather wants the children in his life to see him as a giant, a genius, a god. I could only imagine how difficult this was going to be for my grandfather to explain to my son that times had changed, things had changed, and maybe he wasn’t as sharp as he once was. My grandfather said none of those things. Instead he continued:

“Taylor, your teachers and your school and your principal aren’t going to tell you this, but the truth is when it comes to arithmetic, and I mean real world arithmetic, not the stuff they have picked out for you and put into those books, well the truth is 7’s and 9’s don’t figure. The answer to your homework will never come out right because one of the numbers ends with a 9. So I did the only thing you can do, in real life I mean, I rounded the 9 up to a 10. Sure you can round a 7 or 9 down as well if you want to low-ball it, but I figured this would be easier for you to follow.”

I stared for a moment, incredulous, not sure if my son was believing this, if my grandfather was believing it. I had no earthly idea what to say. Then I thought about my grandfather who in his day had worked as a machinist, who built the die and tool that was used to punch the first removable soda-pop top. Remember, (or maybe you don’t), those first aluminum soda cans that had the tab you just pulled off of the can entirely and chucked it onto the ground? Obviously that was before “give a hoot—don’t pollute” and those pull tabs littered highways everywhere until someone got the idea to make the tab a part of the larger can. Still, discounting the negative environmental aspect, in its day the pop-top was an ingenious piece of engineering. A technological leap and my grandfather was a part of it.

Then I considered how much computers have changed, from the time when I was an 11 year old boy banging out BASIC on my TRS-80, to now when the processor in my wristwatch has more memory and operating capacity than some of the machines that were remarkably once labeled “personal computers”. Day in and day out at the office, I see the technological envelope pushed here at SoftLayer. We offer our customers the latest and greatest from integrated remote out of band management, to high speed fault tolerant digital backups. I am an integral part of one of the most exciting and talked about technology ventures in the history of webhosting. Yet will there come a day when I am sitting at the table with my own son’s children wondering how it happened that I can’t manage to come up with the correct answer for an elementary school problem?

“Well for now,” I said trying to sound authoritative, “I guess we better do it the way your teacher wants—the way the book explains it. You have quite a while to go before you are out in the real-world and by then I bet you have figured out how things work all on your own.” Taylor shrugged and wrote down 8,832. Grandpa started to speak, hesitated, and then held up his index finger. “Does this look crooked to you Taylor?” he asked. “Say maybe you could help me straighten it out by giving it a little pull?” Laughter ensued.

Around the office I am commonly considered a “low-level” software engineer. If you are in the business of computer programming you know this means I generally have various pieces of computer hardware strewn about my work area, and an ASCII chart hanging on my wall complete with a cheat-table so I can quickly convert numbers between binary, decimal, and hex. If you are not in the business of developing software, think of me as guy who couldn’t decide if I wanted to be an electrical engineer or a computer programmer and thus through my own indecision eventually found myself stuck somewhere in between. I know a bit about both but am not an expert in either. (I think the Roman word for this sort of limbo is purgatory, but I find it pretty cozy most days.)

At any rate when a project comes along that walks the fence between the realms of hardware and software my name naturally comes up. Such was the case a few weeks ago when one of our systems administrators had the need to retrieve the serial number from the RAM chips already installed in a number of servers. He asked me if it could be done. I looked and saw the information was reported in the BIOS of one of my machines, so I promptly responded with a “you bet”. After all, if the BIOS can display the information on the screen I should be able to as well. Right? I told him it would take a week.

The problem in this career field I have worked for some ten years now is you don’t know what you don’t know. Fast forward two weeks. Now think the Friday before Easter. That’s right, the one I am supposed to be off lounging around the house in my pajamas. It took a little longer to pull that serial number than I expected. If you’re interested the slow down turned out to be that the information existed at a physical memory address that was not easily accessible from Microsoft Windows (luckily for the BIOS it gets to display the data before an operating system is loaded).

Remember the old Chevy Chase movie “Funny Farm“? Chevy’s character is driving around lost when he passes the old man sitting on his porch in a rocking chair. Chevy stops his vehicle, rolls down his window, and says: “Excuse me Sir. Can you tell me how you would get to Redbud?” The old man leans forward, spits, and replies: “If I were going to Redbud I sure as hell wouldn’t start from here.”

Like Mr. Chase’s character in the movie, I didn’t get to pick where I started the journey from. We need the data available to us after the operating system boots. So I am hacking my way through it. I’m nearly there now. Close enough at least that I felt comfortable taking a break from the code and blowing off some steam by writing this blog. And the truth is, while I might have been whining just a bit I actually have enjoyed this project immensely. I appreciate the fact that the management here at SoftLayer gives us the opportunity to challenge ourselves and then grow to meet those challenges. We are encouraged to “get our hands dirty”. When I finish up here I will have a deeper understanding of how the BIOS relates to the operating system (and through the BIOS indirectly to the hardware).

As for our customers, well, it just so happens once I got to digging around in the binary mud there was a whole lot of other useful insight buried in the swirls of all those zeros and ones. Instead of extracting just the serial numbers I am pulling about a dozen pages of hardware data points we can use in statistical analysis for predicting failures, standards compliance, and availability trends. Like I said, you don’t know what you don’t know. But sometimes you are pleasantly surprised once you find out. By promoting such an amiable work environment, fostering creativity, and encouraging innovation, SoftLayer continues to boldly go where no other hosting company has gone before.

Alright, time to climb down from the pulpit and finish up my software.

Thanks for listening!

My grandmother used to say an ounce of prevention is worth a pound of cure. Usually this was her polite way of telling me to pick my skateboard up off the stairs before she stepped on it and broke her neck or to put a sheet of newspaper over her antique kitchen table before I began refueling my model airplane. All very sound advice looking back. And now here I find myself repeating the same adage some twenty years later in the context of predicting mechanical drive failure. An ounce of prevention is worth a pound of cure.

Hard disk drive manufacturers recognized both the reality and the advantages of being able to predict normal hard disk failures associated with drive degradation sometime around 2003. This led a number of leading hard disk makers to collaborate on a standard which eventually became known as SMART. This acronym stands for Self-Monitoring, Analysis and Reporting Technology and when used properly is a formidable weapon in any system administrator’s arsenal.

The basic concept is that firmware on the hard disk itself will record and report key “attributes” of that drive which when monitored and analyzed over time can be used to predict and avoid catastrophic hard disk failures. Anyone who has been around computers for more than a day knows the terrible feeling that manifests in the pit of your stomach when it becomes apparent that your server or workstation will not boot because the hard disk has cratered. Luckily, we ALL of course back up our hard drives daily! Right?

All kidding aside even with a recent back up just the task of restoring and getting your system back in working order is a serious hassle and it’s not something you get the luxury of scheduling if the machine is critical to operations and failed in the middle of your work day or worse yet, the middle of your beauty sleep. That is where SMART comes in. When properly used SMART data can give “clues” that a drive is reaching a failure point–prior to it failing. This in turns means you can schedule a drive cloning and replacement within your next regular maintenance window. Really aside from a hard disk that lasts forever what more could an administrator ask for?

SMART drive data has been described as a jigsaw puzzle. That’s because it takes monitoring a myriad of data points consistently over time to be able to put together a picture of your hard disk health. The idea is that an administrator regularly records and analyzes characteristics about the installed spinning media and looks for early warning signs that something is going wrong. While different drives have different data points, some of the key and most common attributes are:

• head flying height
• data throughput performance
• spin-up time
• re-allocated sector count
• seek error rate
• seek time performance
• spin try recount
• drive calibration retry count

These items are considered typical drive health indicators and should be base-lined at drive installation and then monitored for significant degradation. While the experts still disagree on the exact value of SMART data analysis, I have seen sources that claim at least 30% of drive failures can be detected some 60 days prior to the actual failure through the monitoring of SMART data.

Of course not all drive failures can be predicted. Plus some failures are caused by factors other than drive degradation. Consider drives damaged by power surges or drives that are dropped in shipping as good examples of drive failures that cannot normally be detected through SMART monitoring. However in my humble opinion even one hard disk failure prevented over the course of my career is something to celebrate–unless you happen to own stock in McNeil Consumer Healthcare, a.k.a. the distributors of Tylenol!

So what does this have to do with SoftLayer? Well I am certainly not claiming that SoftLayer is going to predict all your hard drive disasters so there is no reason for you to back up your data. In fact, I recommend not just backing it up but backing it up in geographically disparate locations (did I mention we have data centers in Dallas and Seattle?). What I do mean to share is that technologies like SMART data are just one of the many ways SoftLayer is currently investigating to improve what is already the best hosting company in the business.

I should know. I was tasked with writing the low-level software to extract this data. That’s right. SoftLayer has engineers working at the application layer, down at the device driver layer, and everywhere in between. If that doesn’t give you a warm fuzzy about your hosting company, I don’t know what will.

The high-pitched whine of a drill sends a shiver down my spine. I jump a little in my seat at a loud bang followed by shuffling feet and mumbled voices. I involuntarily cower at the unmistakable sound of a saw blade spinning—gaining momentum—biting. Nope, I’m not sitting in a theater watching Eli Roth’s next installment in the Hostel franchise. In fact, I’m at the office.

That’s right. I’m sitting at my desk. Sitting at my desk and trying hard to ignore the plethora of singing power tools and crooning contractors who for the last two months have been busy putting up dry wall, wiring electrical outlets, installing locks, and occasionally setting off the fire alarm. It’s the sound of growth. And at the risk of conjuring up images of bad 80’s haircuts, guys in jeans way-too-tight, and shirts where the collars just wouldn’t seem to stay down– one might dare refer to the ruckus as “growing pains”.

Make no mistake about it, growing is painful. Take it from me. I think I was 19 before I managed to grow enough facial hair to require the use of a razor. Combine that tidbit of info with the fact that I had every 8-bit computer known to man proudly on display in my room right next to my impressive collection of latex Hollywood style monster masks and you’ll start to get the picture. Growing requires a lot of work and allows almost no planning as humans have a habit of blossoming in their own sweet time. Companies are no different.

So while management did everything possible to make the required building expansion as unobtrusive as possible, well, it’s still construction work within earshot of a whole team of developers, technicians, and engineers. That’s just the way it is. And while I may complain about the noise and distractions now and again, there is also something very comforting about knowing that I am working at a place that is growing. Growing phenomenally, in a time when not all technology companies are fairing so well.
When the dust settles there will be a lot of new space.

More space means a lot of new hires. More space means more opportunity for existing employees. And yes, more space means more work for everyone involved. Having worked for three failed ventures in as many years, I can tell you I am more than happy to be putting my time and effort and energies into something that is successful; something that continues to be more successful every day. It feels good to be on the winning team for a change. Hearing what some of the other engineers here are saying I don’t think I’m alone in that sentiment.

That’s not to say I’ll miss the noise when the construction is all said and done. Which in case you are interested sounds to be winding down. As for SoftLayer, well something tells me we are just getting started.