For over a decade IPv6 has been touted as the next generation protocol for the Internet. While IPv4 has served the public well since its inception, as early as 1990 it was clear that IPv4 simply didn’t have enough address space to keep up with the phenomenal growth of the web. So in 1994, the gurus got together and finalized IPv6. But solving a problem on paper, and rolling that solution out across the world-wide, publicly accessible series of interconnected computers known as the Internet, takes time. Despite the inevitability of IPv6, on the whole, only a handful of industry leaders are ready to deliver.

Which brings us to SoftLayer. If you follow web hosting news at all, you’ll know that here at SoftLayer we recently solidified our position on the technology forefront by announcing native IPv6 support across our entire array of data centers. If you’re interested in checking out the complete press release, you can find it here. If you are interested in knowing the nuts and bolts of IPv6, I’d recommend taking a look at the IPv6 information page found here. However, if like me, the real burning question on your mind is: “what ever happened to IPv5?” then look no further my friend. You’ve come to the right place.

Unfortunately, the answer is not nearly as exciting as the question. It seems that in the late seventies, an experimental protocol was developed for the internet community, and that this protocol (known as ST2), got dibs on the magical designator of the number five. ST2, like a lot of inventions in the computer industry, didn’t make it. So I thought as a tribute to IPv6’s fallen comrade, IPv5, I’d list a couple of other computing faux pas. Enjoy!

Apple III

Designed as a business computer and the successor to the popular Apple II, the Apple III was a commercial disaster. With a starting price of over $4K, an operating system with the appropriate acronym SOS, and reports of the machine becoming so hot floppy disks would come out of the slot melted down to putty, the Apple III quickly found its way on the list of products discontinued by Apple Inc.

Atari 400

While the Atari 400 itself was not a total failure, it is best known today as the poster child for how NOT to design a keyboard. Marketed as a durable and spill resistant alternative, the flat, zero feedback, sealed ‘membrane’ keyboard was actually chosen by Atari execs because it was vastly less expensive to manufacture than a traditional keyboard. Not only was it nearly impossible to tell if a key had actually been depressed when typing without looking up at the screen, but the deadly ‘break’ key sat right near the often used backspace key. Hard not to feel sorry for anyone who had to peck out more than a command or two on this bad boy.

Windows ME

Rated 4th in PC World’s top 25 worst tech products of all time lists, the acronym was quickly redubbed around the world from the intended Millennium Edition to Mistake Edition. Users reported problems with installing it, getting it to run, getting it to work with hardware, getting it to work with software, and even getting it to stop running so they could go back and install an OS that did work!

Microsoft BOB

Ever wish instead of a desktop interface you interacted with your computer via a big yellow smiley face? No? Apparently you are not alone–evident by the announcement and subsequent retraction of the 1995 software offering MS BOB. The idea behind BOB was to create a replacement for the Windows interface that would make computing more friendly for the everyday user. A noble idea but one implemented poorly. To date, MS BOB has been Microsoft’s most visible product failure.

Google X

Perhaps the shortest lived and most mysterious on my list is the Google X Site. Google X was nothing more than a search home page, styled after the Mac OS Dock from OS X. There was a quote on the bottom of the page that read: “Roses are red. Violets are blue. OS X rocks. Homage to you.” Exactly one day after its release, Google pulled the page without sighting a reason. Could it be Apple copyright attorneys weren’t so flattered?

The following news article came down the SP Wire on Thursday December 18 @ 05:00 Greenwich Mean Time.

At approximately 11:30 Eastern Time senior agent Donald Bolden of the FBI’s cyber crime division held a press conference at the Langley, VA field office to confirm rumors that a laptop containing a large amount of sensitive information had indeed gone missing from the prominent Kringle Corp entity. The information contained on the laptop, is cited as “personal data” by Bolden, but an unnamed source tells this reporter the missing database is purportedly the infamous “naughty list”. Among other items, the stolen laptop holds the full name, address, and social security number for an estimated 2.2 billion minors from all around the world.

“This heinous act of theft truly has the potential to turn this Christmas holiday upside down,” said Bolden. “We urge the responsible party to turn yourself and the laptop in.” At this point in the investigation, the FBI and Kringle Corp are offering amnesty to anyone who brings the stolen property forward. “Don’t kid yourself though,” Bolden warned. “The FBI will find the culprit in this crime and should the laptop not be recovered until after December 24th CEO and founder of Kringle Corp, world renowned philanthropist Christopher Kringle himself, has agreed to prosecute to the full extent of the law.”

When describing exactly how the laptop went missing details get a little fuzzy. According to company logs, the laptop was last signed for by one Edward Keebler. Keebler worked for Elfco, a wholly owned subsidiary of the Kringle Corp located just outside of Mexico City. The laptop was apparently left in Keebler’s car. Company policy strictly prohibits the taking of any company laptop from the premises and as such, Keebler has been let go.

In a statement from Keebler’s lawyer, Mr. Keebler expressed being “truly sorry” for his actions. He states he was aware of the company policy, but under extreme stress from working long hours at the toy assembly plant. He had intended to use the laptop from home to catch up on some paperwork. He now realizes it was a mistake, he agrees with Kringle’s decision to terminate his employment, and he asks nothing more than to be left alone. Keebler’s lawyer also confirmed that his client would be leaving Mexico and returning to his family’s summer home, where he hopes to maybe go back to work as a pastry decorator in his older brother’s cookie factory.

While Bolden states he believes the thief or thieves probably were not aware of what was on the hard drive of the red and white striped laptop when it was taken, the FBI has not yet ruled out that this was an act of sabotage. “Let’s face it,” said Bolden. “Everybody and their brother wants to be Mr. Kringle. If this Christmas gets wrecked, there are a number of other entities out there who stand to gain from Kringle Corp’s misfortune.” Although Bolden refused to name any of Kringle’s competitors specifically, in a public announcement to shareholders just last April, founder and majority stockholder, E. Bunny, of the East-R conglomerate stood before his fellow stake holders and announced his clear intention to move his company into the pole position by the year 2011, using “whatever means necessary”.

While Christopher Kringle himself was not available for comment, his “right-hand-sized” man, Jack Frost, made a statement on his behalf. “Regardless of the outcome to this series of unfortunate events, Mr. Kringle and his entire organization is dedicated to their vision. If we have to double our call center staff in India , telephone every parent, and re-key all the data, then that’s what we’ll do.” Despite the seriousness of this situation, Mr. Frost remained cheerful. As he hurried along with his personal security entourage from the site of the press conference to his corporate sleigh, he took the time to comfort one somber looking little boy and his mother.

“ You can count on us,” Frost told the teary eyed kid. “We are going to do whatever it takes to insure a Merry Christmas to all, and to all a good night!”

Recently on one of our technical forums I contributed to a discussion about the Windows operating system. One of our director’s saw the post and thought it might be of interest to readers of the InnerLayer as well. The post focused on the pros and cons of Windows 2008 from the viewpoint of a systems / driver engineer (aka me). If you have no technical background, or interest in Microsoft operating system offerings, what follows probably will not be of interest to you—just the same, here is my two cents.

Microsoft is no different than any other developer when it comes to writing software–they get better with each iteration. There is not a person out there who would argue that the world of home computers would have been better off if none of us ever progressed beyond MS-DOS 1.0. Not to say there is anything wrong with MS-DOS. I love it. And still use it occasionally doing embedded work. But my point is that while there have certainly been some false starts along the way (can you say BOB), Microsoft’s operating systems generally get better with each release.

So why not go out and update everything the day the latest and greatest OS hits the shelves? Because as most of you know, there are bugs that have to get worked out. To add to that, the more complex the OS gets, the more bugs there are and the more time it takes to shake those bugs out. Windows Server 2008 is no different. In my experience there are still a number of troublesome issues with W2K8 that need to be addressed. Just to name a few:

• UAC (user access control) – these are the security features that give us so much headache. I’m not saying we don’t need the added security. I’m just saying this is a new arena for MS and they still have a lot to learn. After clicking YES, I REALLY REALLY REALLY WANT TO INSTALL SAID APPLICATION for the 40th time in a day, most administrators will opt to disable UAC, thereby thwarting the added security benefits entirely. If I were running this team at MS I’d require all my developers to take a good hard look at LINUX.
• UMD (user mode drivers) – the idea of running a device driver, or a portion of a device driver, in the restricted and therefore safe user memory of the kernel is a great idea in terms of improving OS reliability. I’ve seen numbers suggesting that as many as 90% of hard OS failures are caused by faulty third-party drivers mucking around in kernel mode. However implementing user mode drivers adds some new complexities if hardware manufacturers don’t want to take a performance hit and from my experience not all hardware vendors are up to speed yet.
• Driver Verification – this to me is the most troublesome and annoying issue right now with the 64-bit only version of W2K8. Only kernel mode software that has been certified in the MS lab is allowed to execute on a production boot of the OS. Period. Since I am writing this on the SoftLayer blog, I am assuming most of you are not selecting hardware and drivers to run on your boxes. We are handling that for you. But let me tell you it’s a pain in the butt to only run third party drivers that have been through the MS quality lab. Besides not being able to run drivers we have developed in house it is impossible for us to apply a patch from even the largest of hardware vendors without waiting on that patch to get submitted to MS and then cleared for the OS. A good example was a problem we ran into with an Intel Enet driver. Here at SoftLayer we found a bug in the driver and after a lot of back and forth with Intel’s Engineers we had a fix in hand. But that fix could not be applied to the W2K8 64-bit boxes until weeks later when the fix finally made it from Intel to MS and back to Intel and us again. Very frustrating.

Okay, so now that you see some of the reasons NOT to use MS Windows Server 2008 what are some of the reasons it’s at least worth taking a look at? Well here are just a few that I know of from some of the work I have done keeping up to speed with the latest driver model.

• Improved Memory Management – W2K8 issues fewer and larger disk I/O’s than its 2003 predecessor. This applies to standard disk fetching, but also paging and even read-aheads. On Windows 2003 it is not uncommon for disk writes to happen in blocks < 64KB and certainly never more than 64KB (this was a limitation that had been in place since Windows NT). On W2K8 the memory manager frequently handles writes as large as a full MB. Furthermore, writing related data in large blocks (especially during paging) significantly reduces fragmentation vastly improving read back times as well.
• Improved Data Reliability – Everyone knows how painful disk corruption can be. And everyone knows taking a server offline on a regular basis to run chkdsk and repair disk corruption is slow. One of the ideal improvements in terms of administering a websever is that W2K8 employs a technology called NTFS self-healing. This new feature built into the file system detects disk corruption on the fly and quarantines that sector, allowing system worker-threads to execute chkdsk like repairs on the corrupted area without taking the rest of the volume offline.
• Scalability – The W2K8 kernel introduces a number of streamlining factors that greatly enhance system wide performance. A minor but significant change to the operating system’s low level timer code, combined with new I/O completion handling, and more efficient thread pool, offer marked improvement on load-heavy server applications. I have read documentation supporting claims that the minimization in CPU synchronization alone results directly in a 30% gain on the number of concurrent Windows 2008 users over 2003. That’s not to say once you throw in all the added security and take the user mode driver hit you won’t be looking at 2003 speeds. I’m just pointing out hard kernel-level improvements that can be directly quantified by multiplying your resources against the number of saved CPU cycles.

Alright, no need to beat a dead horse. My hope was if nothing else to muddy the waters a bit. The majority of posts I read on our internal forums seemed to recommend avoiding W2K8 like the plague. I’m only suggesting while it is certainly not perfect, there are some benefits to at least taking it for a test drive. Besides, with SoftLayer’s handy dandy portal driven OS deployment, in the amount of time it took you to read all my rambling you might have already installed Windows Server 2008 and tried it out for yourself. Okay, maybe that’s a bit of an exaggeration. But still…you get the idea!

A few years ago I injured my arm. I won’t go into details about the stupid things some of us do when we are off work, but the long and the short of it was that I ended up with a broken elbow. The surgery to repair the damage left me with a knot near my elbow. Hardly noticeable, in my opinion, but there if you know what you are looking for.

Not too long after the accident, my son, who was 5 going on 6, asked if he could have a friend spend the night. Sure. I picked the two of them up, loaded them in the back seat, and headed for my house. When we reached the first red light between the school and my house, I snatched my Diet Mountain Dew from the console and took a big swig. I felt a tap on my shoulder. I turned my head. It was my son’s friend.

“Mr. Francis,” he said shyly. I thought I knew what was coming. His mom had been very specific. No caffeine.

“Yes,” I replied quickly tilting the bottle to my lips operating on the premise the best defense was a good offense and if I just drained the soda entirely my problem would be solved.

“Can I touch your meatball, please?”

About then is when the carbonated soda came spewing forth from both nostrils.

“What?” I sputtered, my eyes watering and my nose burning. I checked the rearview mirror certain Chris Hansen from Dateline’s “To Catch a Predator” was going to smiling at me from the backseat along with an entire NBC camera crew. There was no Chris Hansen. Just my son and his school buddy giggling.

“Your meatball,” the kid said, pointing to the bump near my elbow. My own child nodded enthusiastically.

Ah, now I understood. There had simply been a miscommunication.

Certain the last thing I needed was some kid going home telling his parents Mr. Francis let him touch his meatball, I politely told him not only could he not touch my meatball but it would be best if we didn’t talk about my meatball at all. Both boys seemed mildly disappointed but quickly got over it when I suggested we make a detour for the nearest McDonald’s.

Fast forward to a couple weeks ago when we had our monthly development meeting here in the SoftLayer headquarters facility. Our VP of Development, Matt Chilek, gave us a talk about the importance of clear and concise communications. Specifically error messages in the portal.

The SoftLayer customer portal is probably the most sophisticated tool of its kind for remote management of servers. So no matter how much testing we do internally, now and again an error will pop up. Sometimes, these errors are legitimate bugs. Other times, they are runtime issues, such as a temporary outage of a database or some support hardware. In either case, how we present the error to the customer is of the utmost importance.

I’ll give you an example. The first time I worked on the WSUS update page in the portal, if my application failed to get a response from the MS Windows Update Server I threw up an error message: “fatal error”. Which is accurate. Sort of. The error is fatal to the application at that particular time. But that doesn’t really give the customer or our datacenter technicians a lot to go on. A better error message is “No response from WSUS server @192.100.12.1. This server could be temporarily offline for maintenance or updates. Please try again in a few minutes. If the problem persists contact technical support.”

While both error messages alert us that something went wrong, the second lets us know what the error was. Exactly which hardware was the culprit. And that the issue might only be temporary so give it a few minutes before crying that the sky is falling. Clear. Concise. To the point. That is the only way to keep a tool as complex and feature rich as the SoftLayer portal from overwhelming our customers and employees alike.

So the SoftLayer development team is making a concerted effort to do just that. And we could really use the help of SoftLayer employees from other departments as well as our customers who use the portal on a regular basis, in pointing out any areas where the language used or information presented is not as clear as it could be. It only takes a minute to fill out a ticket with a note to the dev team, and, in the end, it is you who will benefit.

Alright, I suppose I should get back to writing code instead of writing about writing code. But first I think I’ll make a quick trip to the employee break room to grab some caffeine. And if by chance you run into me in the hallway, no you can’t touch my meatball—so don’t even ask.

A couple weeks ago, I made a quick stop by my friendly neighborhood drug store to pick up some film for my camera. When I came out of the store, I hopped in my car, turned the key, and then… BOOM! That’s right—there was an explosion. Nothing earth shattering, but alarming none-the-less. The explosion was certainly loud enough to turn a few heads. And it gave me a bit of a scare as there was a moment where I found myself wondering if my ex wife had finally saved up enough money to have me taken out.

After giving the smoke a few minutes to clear out, and my heart a few beats to find its way out of my throat and back into my chest, I got out and looked under the hood. Not because I know anything about what makes a car tick, but because looking under hoods is what guys do. Especially when people are watching. In fact, another guy nearby came over and looked under the hood too. And as soon as he opened his mouth I realized that like me, he was only looking under the hood because that is what guys do in these situations.

“I don’t think your battery is supposed to have a big crack down the middle like that,” he said to me.

“Me neither,” I mumbled. I suppressed the urge to ask Mr. Obvious where he went to automotive school.

“You try it again?” he asked me.

For a second I thought he was joking. Then I realized he wasn’t and more importantly that I didn’t have any better ideas. So I hunted around the engine block until I found where the top of the battery case had been propelled, just sort of laid it over the smoking remnants of the battery, then slid back into the car and tried the key. It didn’t start. When I turned the key all the lights came on, all the dials and gauges swung wildly from side to side, and then everything went dark. Lights out. I tried again. But this time there weren’t even any lights. My sporty Mazda 6 might as well have turned into a pumpkin for all the good it was going to be getting me home. So much for zoom, zoom, zoom.

“You might need to call for a tow,” said Nostradamus now standing by my car window with his hands in his pockets.

“Thanks again,” I said unenthusiastically, “I got it from here.” I hoped he’d take the hint.

He did.

I waited till I saw him drive off then tried the key one more time. Nothing. So I broke down and called the towing service. Rather than calling a friend or a taxi, I opted to walk the two miles home from the drug store. During which I had some time to think. It occurred to me that the idea I might start up my car and drive it to the nearest shop for diagnostics after it just got done blowing up in my face was pretty outlandish. And yet, that is exactly what we offer with the servers we sell at SoftLayer.

Lights Out Management (LOM) or Out Of Band Management (OOB) as it is sometimes called is a feature we include with all of our servers at SoftLayer. If you’re a current customer, you have probably noticed the “management ip address” noted for your servers in our portal. That’s exactly what I’m referring to. And while LOM is the stuff of science fiction in automobiles, in our world class servers it’s a reality. That’s right, with our OOB offerings you can:

• Cycle the power on a server even if the operating system has crashed, locked, or otherwise blown up.
• Start up a dead server with the push of a button.
• Get critical readings of system health indicators like processor temperature and fan speeds, regardless of what operating system or software you have installed on that server.
• Manipulate system BIOS and perform diagnostics remotely with full video, keyboard, and mouse support.
• And a whole slew of other things that will make your life much much easier.

In essence, SoftLayer’s OOB management features are the next best thing to driving to one of our data centers and plugging a keyboard into your server. Maybe even better, since you don’t have to fight the traffic. It’s the sort of thing a system administrator dreams of. The sort of thing that sets SoftLayer apart from the myriad of other hosting companies out there.

As for my car, two days and two hundred dollars later it was back in my driveway. Apparently the mechanic who worked on my vehicle went to the same school of engineering as the Good Samaritan who provided me so much insight two days prior. The diagnosis, according to the invoice, was: cracked battery. Then just to give me a warm and fuzzy written in big block letters at the bottom of the page was:

IF THIS HAPPENS AGAIN BRING IT BACK IN

I can’t wait till the automotive industry catches up to SoftLayer.

Through the virtue of me having young parents, at age nine my own son Taylor has the fortune of not just having grandparents, but great grandparents alive and well. On my mother’s side in particular, I have a grandfather (after who I am named), who is still quite the traveler at age 72. While he lives in Ohio he frequently “pops in” on my son and me. Despite his inability to call and let me know he is coming or how long he will stay, once I get over the initial shock of discovering he is waiting in the driveway for me to come home from work and welcome him into my home, we usually do have a nice visit. (Though he has yet to convince me to give him his own garage door opener despite asking on more than one occasion!)

My son especially likes having his great-grandfather around. My grandfather, as I am sure do most grandfathers, has seventy plus years worth of stories and opinions and riddles he has collected over a remarkably varied life. And if there is one thing that my grandfather is not, it is shy; so my son finds his great grandpa immensely entertaining–as did I at age nine. (Although between you and me I really thought by the time I was nine I had stopped falling for the old pull-my-finger trick that still sends my grandfather and my son into fits of laughter visit after visit.)

The last time my grandfather came out to visit, Taylor had a lot of homework. So after dinner my grandfather settled onto the couch to watch wrestling, (pronounced WRAST’lin), and Taylor and I went about trying to do his homework. The assignment was geared at reinforcing multiplication tables. Something my son struggled with for a bit. So we were working on it for a while. Long enough that my grandfather decided either the school was passing out too much homework, or I was explaining it wrong. So like any concerned great grandparent would do my grandfather clicked off the TV, walked into the kitchen, and pulled up a chair intent on showing us the error of our ways.

Grandpa asked Taylor to explain the assignment–which my son did. Without warning Grandpa then plucked the page and pencil from my son’s hands and proceeded to stare over a multiplication problem for some time. The page stared back at him.

128 x 69 =

Taylor and I watched with growing fascination as grandpa proceeded to scribble nearly as many figures on the page as there were problems. At last he grunted and wrote his answer.

8960

Now I am by no means a mathematical giant, but something seemed a bit peculiar about his answer. So I did a quick computation and came up with 8,832. And while I was still trying to politely figure out how to tell my grandpa “thanks but no thanks”, my son didn’t show any such discretion.

“That’s wrong Great Grandpa!” he exclaimed.

My grandfather took the page back, made some more of his calculations in the margin, then looked up in all seriousness and said to my son:

“Taylor, you are old enough to know the truth.”

I have to tell you at this moment I was pretty shocked. While I was not sure what great personal revelation my grandfather intended to make, I was sure it was to be a difficult one. Every father and grandfather and great grandfather wants the children in his life to see him as a giant, a genius, a god. I could only imagine how difficult this was going to be for my grandfather to explain to my son that times had changed, things had changed, and maybe he wasn’t as sharp as he once was. My grandfather said none of those things. Instead he continued:

“Taylor, your teachers and your school and your principal aren’t going to tell you this, but the truth is when it comes to arithmetic, and I mean real world arithmetic, not the stuff they have picked out for you and put into those books, well the truth is 7’s and 9’s don’t figure. The answer to your homework will never come out right because one of the numbers ends with a 9. So I did the only thing you can do, in real life I mean, I rounded the 9 up to a 10. Sure you can round a 7 or 9 down as well if you want to low-ball it, but I figured this would be easier for you to follow.”

I stared for a moment, incredulous, not sure if my son was believing this, if my grandfather was believing it. I had no earthly idea what to say. Then I thought about my grandfather who in his day had worked as a machinist, who built the die and tool that was used to punch the first removable soda-pop top. Remember, (or maybe you don’t), those first aluminum soda cans that had the tab you just pulled off of the can entirely and chucked it onto the ground? Obviously that was before “give a hoot—don’t pollute” and those pull tabs littered highways everywhere until someone got the idea to make the tab a part of the larger can. Still, discounting the negative environmental aspect, in its day the pop-top was an ingenious piece of engineering. A technological leap and my grandfather was a part of it.

Then I considered how much computers have changed, from the time when I was an 11 year old boy banging out BASIC on my TRS-80, to now when the processor in my wristwatch has more memory and operating capacity than some of the machines that were remarkably once labeled “personal computers”. Day in and day out at the office, I see the technological envelope pushed here at SoftLayer. We offer our customers the latest and greatest from integrated remote out of band management, to high speed fault tolerant digital backups. I am an integral part of one of the most exciting and talked about technology ventures in the history of webhosting. Yet will there come a day when I am sitting at the table with my own son’s children wondering how it happened that I can’t manage to come up with the correct answer for an elementary school problem?

“Well for now,” I said trying to sound authoritative, “I guess we better do it the way your teacher wants—the way the book explains it. You have quite a while to go before you are out in the real-world and by then I bet you have figured out how things work all on your own.” Taylor shrugged and wrote down 8,832. Grandpa started to speak, hesitated, and then held up his index finger. “Does this look crooked to you Taylor?” he asked. “Say maybe you could help me straighten it out by giving it a little pull?” Laughter ensued.

Around the office I am commonly considered a “low-level” software engineer. If you are in the business of computer programming you know this means I generally have various pieces of computer hardware strewn about my work area, and an ASCII chart hanging on my wall complete with a cheat-table so I can quickly convert numbers between binary, decimal, and hex. If you are not in the business of developing software, think of me as guy who couldn’t decide if I wanted to be an electrical engineer or a computer programmer and thus through my own indecision eventually found myself stuck somewhere in between. I know a bit about both but am not an expert in either. (I think the Roman word for this sort of limbo is purgatory, but I find it pretty cozy most days.)

At any rate when a project comes along that walks the fence between the realms of hardware and software my name naturally comes up. Such was the case a few weeks ago when one of our systems administrators had the need to retrieve the serial number from the RAM chips already installed in a number of servers. He asked me if it could be done. I looked and saw the information was reported in the BIOS of one of my machines, so I promptly responded with a “you bet”. After all, if the BIOS can display the information on the screen I should be able to as well. Right? I told him it would take a week.

The problem in this career field I have worked for some ten years now is you don’t know what you don’t know. Fast forward two weeks. Now think the Friday before Easter. That’s right, the one I am supposed to be off lounging around the house in my pajamas. It took a little longer to pull that serial number than I expected. If you’re interested the slow down turned out to be that the information existed at a physical memory address that was not easily accessible from Microsoft Windows (luckily for the BIOS it gets to display the data before an operating system is loaded).

Remember the old Chevy Chase movie “Funny Farm“? Chevy’s character is driving around lost when he passes the old man sitting on his porch in a rocking chair. Chevy stops his vehicle, rolls down his window, and says: “Excuse me Sir. Can you tell me how you would get to Redbud?” The old man leans forward, spits, and replies: “If I were going to Redbud I sure as hell wouldn’t start from here.”

Like Mr. Chase’s character in the movie, I didn’t get to pick where I started the journey from. We need the data available to us after the operating system boots. So I am hacking my way through it. I’m nearly there now. Close enough at least that I felt comfortable taking a break from the code and blowing off some steam by writing this blog. And the truth is, while I might have been whining just a bit I actually have enjoyed this project immensely. I appreciate the fact that the management here at SoftLayer gives us the opportunity to challenge ourselves and then grow to meet those challenges. We are encouraged to “get our hands dirty”. When I finish up here I will have a deeper understanding of how the BIOS relates to the operating system (and through the BIOS indirectly to the hardware).

As for our customers, well, it just so happens once I got to digging around in the binary mud there was a whole lot of other useful insight buried in the swirls of all those zeros and ones. Instead of extracting just the serial numbers I am pulling about a dozen pages of hardware data points we can use in statistical analysis for predicting failures, standards compliance, and availability trends. Like I said, you don’t know what you don’t know. But sometimes you are pleasantly surprised once you find out. By promoting such an amiable work environment, fostering creativity, and encouraging innovation, SoftLayer continues to boldly go where no other hosting company has gone before.

Alright, time to climb down from the pulpit and finish up my software.

Thanks for listening!

My grandmother used to say an ounce of prevention is worth a pound of cure. Usually this was her polite way of telling me to pick my skateboard up off the stairs before she stepped on it and broke her neck or to put a sheet of newspaper over her antique kitchen table before I began refueling my model airplane. All very sound advice looking back. And now here I find myself repeating the same adage some twenty years later in the context of predicting mechanical drive failure. An ounce of prevention is worth a pound of cure.

Hard disk drive manufacturers recognized both the reality and the advantages of being able to predict normal hard disk failures associated with drive degradation sometime around 2003. This led a number of leading hard disk makers to collaborate on a standard which eventually became known as SMART. This acronym stands for Self-Monitoring, Analysis and Reporting Technology and when used properly is a formidable weapon in any system administrator’s arsenal.

The basic concept is that firmware on the hard disk itself will record and report key “attributes” of that drive which when monitored and analyzed over time can be used to predict and avoid catastrophic hard disk failures. Anyone who has been around computers for more than a day knows the terrible feeling that manifests in the pit of your stomach when it becomes apparent that your server or workstation will not boot because the hard disk has cratered. Luckily, we ALL of course back up our hard drives daily! Right?

All kidding aside even with a recent back up just the task of restoring and getting your system back in working order is a serious hassle and it’s not something you get the luxury of scheduling if the machine is critical to operations and failed in the middle of your work day or worse yet, the middle of your beauty sleep. That is where SMART comes in. When properly used SMART data can give “clues” that a drive is reaching a failure point–prior to it failing. This in turns means you can schedule a drive cloning and replacement within your next regular maintenance window. Really aside from a hard disk that lasts forever what more could an administrator ask for?

SMART drive data has been described as a jigsaw puzzle. That’s because it takes monitoring a myriad of data points consistently over time to be able to put together a picture of your hard disk health. The idea is that an administrator regularly records and analyzes characteristics about the installed spinning media and looks for early warning signs that something is going wrong. While different drives have different data points, some of the key and most common attributes are:

• head flying height
• data throughput performance
• spin-up time
• re-allocated sector count
• seek error rate
• seek time performance
• spin try recount
• drive calibration retry count

These items are considered typical drive health indicators and should be base-lined at drive installation and then monitored for significant degradation. While the experts still disagree on the exact value of SMART data analysis, I have seen sources that claim at least 30% of drive failures can be detected some 60 days prior to the actual failure through the monitoring of SMART data.

Of course not all drive failures can be predicted. Plus some failures are caused by factors other than drive degradation. Consider drives damaged by power surges or drives that are dropped in shipping as good examples of drive failures that cannot normally be detected through SMART monitoring. However in my humble opinion even one hard disk failure prevented over the course of my career is something to celebrate–unless you happen to own stock in McNeil Consumer Healthcare, a.k.a. the distributors of Tylenol!

So what does this have to do with SoftLayer? Well I am certainly not claiming that SoftLayer is going to predict all your hard drive disasters so there is no reason for you to back up your data. In fact, I recommend not just backing it up but backing it up in geographically disparate locations (did I mention we have data centers in Dallas and Seattle?). What I do mean to share is that technologies like SMART data are just one of the many ways SoftLayer is currently investigating to improve what is already the best hosting company in the business.

I should know. I was tasked with writing the low-level software to extract this data. That’s right. SoftLayer has engineers working at the application layer, down at the device driver layer, and everywhere in between. If that doesn’t give you a warm fuzzy about your hosting company, I don’t know what will.

The high-pitched whine of a drill sends a shiver down my spine. I jump a little in my seat at a loud bang followed by shuffling feet and mumbled voices. I involuntarily cower at the unmistakable sound of a saw blade spinning—gaining momentum—biting. Nope, I’m not sitting in a theater watching Eli Roth’s next installment in the Hostel franchise. In fact, I’m at the office.

That’s right. I’m sitting at my desk. Sitting at my desk and trying hard to ignore the plethora of singing power tools and crooning contractors who for the last two months have been busy putting up dry wall, wiring electrical outlets, installing locks, and occasionally setting off the fire alarm. It’s the sound of growth. And at the risk of conjuring up images of bad 80’s haircuts, guys in jeans way-too-tight, and shirts where the collars just wouldn’t seem to stay down– one might dare refer to the ruckus as “growing pains”.

Make no mistake about it, growing is painful. Take it from me. I think I was 19 before I managed to grow enough facial hair to require the use of a razor. Combine that tidbit of info with the fact that I had every 8-bit computer known to man proudly on display in my room right next to my impressive collection of latex Hollywood style monster masks and you’ll start to get the picture. Growing requires a lot of work and allows almost no planning as humans have a habit of blossoming in their own sweet time. Companies are no different.

So while management did everything possible to make the required building expansion as unobtrusive as possible, well, it’s still construction work within earshot of a whole team of developers, technicians, and engineers. That’s just the way it is. And while I may complain about the noise and distractions now and again, there is also something very comforting about knowing that I am working at a place that is growing. Growing phenomenally, in a time when not all technology companies are fairing so well.
When the dust settles there will be a lot of new space.

More space means a lot of new hires. More space means more opportunity for existing employees. And yes, more space means more work for everyone involved. Having worked for three failed ventures in as many years, I can tell you I am more than happy to be putting my time and effort and energies into something that is successful; something that continues to be more successful every day. It feels good to be on the winning team for a change. Hearing what some of the other engineers here are saying I don’t think I’m alone in that sentiment.

That’s not to say I’ll miss the noise when the construction is all said and done. Which in case you are interested sounds to be winding down. As for SoftLayer, well something tells me we are just getting started.